Found 200 jobs Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log Found 66 distinct failure reasons Dead: b'2024-04-01T23:00:27.011 DEBUG:teuthology.exit:Finished running handlers' ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7ff8b7054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7ff8b70a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55fd88e17fa5] 6: ceph-osd(+0x3f3106) [0x55fd88e18106] 7: ceph-osd(+0x3e4d9c) [0x55fd88e09d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55fd8937b6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55fd8938571d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55fd893f5061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55fd893f868b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55fd893f9268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55fd893e6fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55fd893e7e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55fd892e770d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55fd890a9592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55fd89071607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55fd89067e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55fd88fa3d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd891eda19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55fd88fbe727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55fd894bcd2a] 23: ceph-osd(+0xa982d4) [0x55fd894bd2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7ff8b709f802] 25: /lib64/libc.so.6(+0x3f450) [0x7ff8b703f450] 3 jobs: ['7626119', '7626137', '7626048'] suites intersection: ['objectstore/bluestore-comp-lz4', 'rados'] suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/none', 'thrashosds-health', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55bf1c4d2fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f793d254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f793d2a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55bf1c4d2fa5] 6: ceph-osd(+0x3f3106) [0x55bf1c4d3106] 7: ceph-osd(+0x3e4d9c) [0x55bf1c4c4d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55bf1ca366c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55bf1ca4071d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55bf1cab0061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55bf1cab368b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55bf1cab4268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55bf1caa1fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55bf1caa2e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55bf1c9a270d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55bf1c764592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55bf1c72c607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55bf1c722e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55bf1c65ed27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55bf1c8a8a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55bf1c679727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55bf1cb77d2a] 23: ceph-osd(+0xa982d4) [0x55bf1cb782d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f793d29f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f793d23f450] 1 jobs: ['7626148'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-snaps-balanced}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x560b19ae6fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1ab0054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f1ab00a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x560b19ae6fa5] 6: ceph-osd(+0x3f3106) [0x560b19ae7106] 7: ceph-osd(+0x3e4d9c) [0x560b19ad8d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x560b1a04a6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x560b1a05471d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x560b1a0c4061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x560b1a0c768b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x560b1a0c8268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x560b1a0b5fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x560b1a0b6e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x560b19fb670d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x560b19d78592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x560b19d40607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x560b19d36e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x560b19c72d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x560b19ebca19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x560b19c8d727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x560b1a18bd2a] 23: ceph-osd(+0xa982d4) [0x560b1a18c2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f1ab009f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f1ab003f450] 1 jobs: ['7626130'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55f2faccefa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f2769454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f27694a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55f2faccefa5] 6: ceph-osd(+0x3f3106) [0x55f2faccf106] 7: ceph-osd(+0x3e4d9c) [0x55f2facc0d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55f2fb2326c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55f2fb23c71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55f2fb2ac061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55f2fb2af68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55f2fb2b0268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55f2fb29dfe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55f2fb29ee33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55f2fb19e70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55f2faf60592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55f2faf28607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55f2faf1ee65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55f2fae5ad27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f2fb0a4a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55f2fae75727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55f2fb373d2a] 23: ceph-osd(+0xa982d4) [0x55f2fb3742d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f276949f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f276943f450] 1 jobs: ['7626075'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-snaps}'] Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 1 jobs: ['7626135'] suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x561c02dc4fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fa94aa54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fa94aaa154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x561c02dc4fa5] 6: ceph-osd(+0x3f3106) [0x561c02dc5106] 7: ceph-osd(+0x3e4d9c) [0x561c02db6d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x561c033286c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x561c0333271d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x561c033a2061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x561c033a568b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x561c033a6268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x561c03393fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x561c03394e33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x561c032ad2a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x561c03292397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x561c030cca56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x561c03016edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x561c02f50d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561c0319aa19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x561c02f6b727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x561c03469d2a] 23: ceph-osd(+0xa982d4) [0x561c0346a2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7fa94aa9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fa94aa3f450] 1 jobs: ['7626074'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/cache-snaps-balanced}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e95f96ffa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8821e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8821ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e95f96ffa5] 6: ceph-osd(+0x3f3106) [0x55e95f970106] 7: ceph-osd(+0x3e4d9c) [0x55e95f961d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e95fed36c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55e95fedd71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55e95ff4d061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e95ff5068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e95ff51268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55e95ff3efe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55e95ff3fe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55e95fe3f70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55e95fc01592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55e95fbc9607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55e95fbbfe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e95fafbd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e95fd45a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55e95fb16727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55e960014d2a] 23: ceph-osd(+0xa982d4) [0x55e9600152d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f8821e9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f8821e3f450] 1 jobs: ['7626070'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/cache-pool-snaps}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5594cc35a0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2d2c865520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5594cc35a0bc] 6: ceph-osd(+0x3f721d) [0x5594cc35a21d] 7: ceph-osd(+0x3e9888) [0x5594cc34c888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5594cc8a4c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5594cc8b905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5594cc91a2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5594cc92352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5594cc9240a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5594cc928611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5594cc929433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5594cc81bc9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5594cc5d1462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5594cc5a796f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5594cc5ab4e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5594cc4e2aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5594cc727799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5594cc4ed793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5594cc9e6313] 23: ceph-osd(+0xa83874) [0x5594cc9e6874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d2c8b7b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d2c949a00] 1 jobs: ['7626150'] suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x563420a4d0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f27ce7e3520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x563420a4d0bc] 6: ceph-osd(+0x3f721d) [0x563420a4d21d] 7: ceph-osd(+0x3e9888) [0x563420a3f888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x563420f97c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x563420fac05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x56342100d2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56342101652b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5634210170a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x56342101b611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x56342101c433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x563420f0ae3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x563420f0c937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x563420d296f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x563420c8174d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x563420bd5aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563420e1a799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563420be0793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5634210d9313] 23: ceph-osd(+0xa83874) [0x5634210d9874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f27ce835b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f27ce8c7a00] 1 jobs: ['7626121'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55b355e280bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0426777520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55b355e280bc] 6: ceph-osd(+0x3f721d) [0x55b355e2821d] 7: ceph-osd(+0x3e9888) [0x55b355e1a888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55b356372c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55b35638705e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55b3563e82a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55b3563f152b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55b3563f20a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55b3563f6611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55b3563f7433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55b3562e9c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55b35609f462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55b35607596f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55b3560794e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55b355fb0aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b3561f5799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55b355fbb793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55b3564b4313] 23: ceph-osd(+0xa83874) [0x55b3564b4874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f04267c9b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f042685ba00] 1 jobs: ['7626023'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55796bafffa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f277f454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f277f4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55796bafffa5] 6: ceph-osd(+0x3f3106) [0x55796bb00106] 7: ceph-osd(+0x3e4d9c) [0x55796baf1d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55796c0636c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55796c06d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55796c0dd061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55796c0e068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55796c0e1268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55796c0cefe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55796c0cfe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55796bfcf70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55796bd91592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55796bd59607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55796bd4fe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55796bc8bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55796bed5a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55796bca6727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55796c1a4d2a] 23: ceph-osd(+0xa982d4) [0x55796c1a52d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f277f49f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f277f43f450] 1 jobs: ['7626129'] suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/rados_api_tests}'] Failure: "2024-04-01T11:14:54.456109+0000 mon.smithi080 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log 8 jobs: ['7626125', '7626152', '7626072', '7626093', '7626177', '7626199', '7626042', '7626012'] suites intersection: [] suites union: ['0-nvme-loop', '0-start', '1-start', '1-tasks', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', 'mon_election/connectivity', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5597371c00bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0badbb9520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5597371c00bc] 6: ceph-osd(+0x3f721d) [0x5597371c021d] 7: ceph-osd(+0x3e9888) [0x5597371b2888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55973770ac30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55973771f05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5597377802a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55973778952b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55973778a0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55973778e611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55973778f433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55973767de3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55973767f937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55973749c6f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x5597373f474d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x559737348aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55973758d799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x559737353793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55973784c313] 23: ceph-osd(+0xa83874) [0x55973784c874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0badc0bb43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f0badc9da00] 1 jobs: ['7626100'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/redirect}'] Dead: b'2024-04-01T21:56:21.199 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['7626010'] suites: ['mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{ubuntu_latest}}'] Failure: No module named 'tasks.ceph' 1 jobs: ['7626204'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ccd46a70bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8ef20eb520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ccd46a70bc] 6: ceph-osd(+0x3f721d) [0x55ccd46a721d] 7: ceph-osd(+0x3e9888) [0x55ccd4699888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ccd4bf1c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55ccd4c0605e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55ccd4c672a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ccd4c7052b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ccd4c710a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55ccd4c75611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55ccd4c76433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55ccd4b64e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55ccd4b66937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55ccd49836f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55ccd48db74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ccd482faa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ccd4a74799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55ccd483a793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55ccd4d33313] 23: ceph-osd(+0xa83874) [0x55ccd4d33874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f8ef213db43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f8ef21cfa00] 1 jobs: ['7626046'] suites: ['mon_election/classic', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/dmclock_1Shard_16Threads', 'settings/optimized', 'ubuntu_latest', 'workloads/radosbench_4M_seq_read}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5621760280bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f976bd45520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5621760280bc] 6: ceph-osd(+0x3f721d) [0x56217602821d] 7: ceph-osd(+0x3e9888) [0x56217601a888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x562176572c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x56217658705e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5621765e82a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5621765f152b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5621765f20a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5621765f6611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5621765f7433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5621764e9c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x56217629f462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x56217627596f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5621762794e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5621761b0aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5621763f5799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5621761bb793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5621766b4313] 23: ceph-osd(+0xa83874) [0x5621766b4874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f976bd97b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f976be29a00] 1 jobs: ['7626126'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x559b630affa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8cf7454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8cf74a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x559b630affa5] 6: ceph-osd(+0x3f3106) [0x559b630b0106] 7: ceph-osd(+0x3e4d9c) [0x559b630a1d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x559b636136c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x559b6361d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x559b6368d061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x559b6369068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x559b63691268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x559b6367efe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x559b6367fe33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x559b635982a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x559b6357d397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x559b633b7a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x559b63301edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x559b6323bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x559b63485a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x559b63256727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x559b63754d2a] 23: ceph-osd(+0xa982d4) [0x559b637552d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f8cf749f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f8cf743f450] 1 jobs: ['7626032'] suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55732de3c0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7feb548af520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55732de3c0bc] 6: ceph-osd(+0x3f721d) [0x55732de3c21d] 7: ceph-osd(+0x3e9888) [0x55732de2e888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55732e386c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55732e39b05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55732e3fc2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55732e40552b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55732e4060a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55732e40a611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55732e40b433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55732e2fdc9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55732e0b3462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55732e08996f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55732e08d4e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55732dfc4aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55732e209799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55732dfcf793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55732e4c8313] 23: ceph-osd(+0xa83874) [0x55732e4c8874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7feb54901b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7feb54993a00] 1 jobs: ['7626176'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/redirect}'] Failure: Command failed on smithi070 with status 5: 'sudo systemctl stop ceph-966734f0-f01a-11ee-b647-cb9ed24678a4@mon.a' 1 jobs: ['7626140'] suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55888501d0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ffa87b9e520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55888501d0bc] 6: ceph-osd(+0x3f721d) [0x55888501d21d] 7: ceph-osd(+0x3e9888) [0x55888500f888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x558885567c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55888557c05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5588855dd2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5588855e652b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5588855e70a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5588855eb611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5588855ec433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x5588854dae3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x5588854dc937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x5588852f96f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55888525174d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5588851a5aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5588853ea799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5588851b0793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5588856a9313] 23: ceph-osd(+0xa83874) [0x5588856a9874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ffa87bf0b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ffa87c82a00] 1 jobs: ['7626105'] suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/many', 'workloads/pool-create-delete}'] Assertion: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x4faf4a] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x4faf4a] 2: ceph-osd(+0x3f3106) [0x4fb106] 3: ceph-osd(+0x3e4d9c) [0x4ecd9c] 4: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0xa5e6c0] 5: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0xa6871d] 6: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0xad8061] 7: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0xadb68b] 8: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0xadc268] 9: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0xac9fe5] 10: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0xacae33] 11: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x9ca70d] 12: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x78c592] 13: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x754607] 14: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x74ae65] 15: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x686d27] 16: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8d0a19] 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6a1727] 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xb9fd2a] 19: ceph-osd(+0xa982d4) [0xba02d4] 20: /lib64/libc.so.6(+0x9f802) [0x54eb802] 1 jobs: ['7626021'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'validater/valgrind}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56426a43cfa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f7557054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f75570a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56426a43cfa5] 6: ceph-osd(+0x3f3106) [0x56426a43d106] 7: ceph-osd(+0x3e4d9c) [0x56426a42ed9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x56426a9a06c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56426a9aa71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x56426aa1a061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56426aa1d68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56426aa1e268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x56426aa0bfe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x56426aa0ce33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x56426a90c70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x56426a6ce592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x56426a696607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x56426a68ce65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56426a5c8d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56426a812a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56426a5e3727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56426aae1d2a] 23: ceph-osd(+0xa982d4) [0x56426aae22d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f755709f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f755703f450] 1 jobs: ['7626077'] suites: ['cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55db39737fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f0896e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f0896ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55db39737fa5] 6: ceph-osd(+0x3f3106) [0x55db39738106] 7: ceph-osd(+0x3e4d9c) [0x55db39729d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55db39c9b6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55db39ca571d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55db39d15061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55db39d1868b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55db39d19268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55db39d06fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55db39d07e33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x55db39c202a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55db39c05397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55db39a3fa56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x55db39989edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55db398c3d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55db39b0da19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55db398de727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55db39ddcd2a] 23: ceph-osd(+0xa982d4) [0x55db39ddd2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f0896e9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f0896e3f450] 1 jobs: ['7626029'] suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/repair_test}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5639750a3fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f685d054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f685d0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5639750a3fa5] 6: ceph-osd(+0x3f3106) [0x5639750a4106] 7: ceph-osd(+0x3e4d9c) [0x563975095d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5639756076c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56397561171d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x563975681061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56397568468b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x563975685268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x563975672fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x563975673e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x56397557370d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x563975335592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5639752fd607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5639752f3e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56397522fd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563975479a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56397524a727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x563975748d2a] 23: ceph-osd(+0xa982d4) [0x5639757492d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f685d09f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f685d03f450] 1 jobs: ['7626106'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/redirect_promote_tests}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56413f9c5fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1aa6454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f1aa64a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56413f9c5fa5] 6: ceph-osd(+0x3f3106) [0x56413f9c6106] 7: ceph-osd(+0x3e4d9c) [0x56413f9b7d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x56413ff296c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56413ff3371d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x56413ffa3061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56413ffa668b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56413ffa7268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x56413ff94fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x56413ff95e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x56413fe9570d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x56413fc57592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x56413fc1f607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x56413fc15e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56413fb51d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56413fd9ba19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56413fb6c727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56414006ad2a] 23: ceph-osd(+0xa982d4) [0x56414006b2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f1aa649f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f1aa643f450] 1 jobs: ['7626144'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/cache-pool-snaps}'] Failure: wait_for_clean: failed before timeout expired 2 jobs: ['7626082', '7626160'] suites intersection: ['debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}'] suites union: ['debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-snappy}', 'random-objectstore$/{bluestore-comp-zlib}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/insights}', 'tasks/per_module_finisher_stats}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55956db7a0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f04560b6520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55956db7a0bc] 6: ceph-osd(+0x3f721d) [0x55956db7a21d] 7: ceph-osd(+0x3e9888) [0x55956db6c888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55956e0c4c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55956e0d905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55956e13a2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55956e14352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55956e1440a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55956e148611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55956e149433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55956e03bc9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55956ddf1462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55956ddc796f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55956ddcb4e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55956dd02aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55956df47799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55956dd0d793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55956e206313] 23: ceph-osd(+0xa83874) [0x55956e206874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0456108b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f045619aa00] 1 jobs: ['7626033'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/redirect_set_object}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560af16fa0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5e32d1e520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560af16fa0bc] 6: ceph-osd(+0x3f721d) [0x560af16fa21d] 7: ceph-osd(+0x3e9888) [0x560af16ec888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x560af1c44c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x560af1c5905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x560af1cba2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x560af1cc352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x560af1cc40a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x560af1cc8611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x560af1cc9433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x560af1bb7e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x560af1bb9937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x560af19d66f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x560af192e74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x560af1882aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x560af1ac7799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560af188d793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x560af1d86313] 23: ceph-osd(+0xa83874) [0x560af1d86874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5e32d70b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5e32e02a00] 1 jobs: ['7626142'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}'] Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 1 jobs: ['7626112'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/valgrind}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55886f233fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f4792454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f47924a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55886f233fa5] 6: ceph-osd(+0x3f3106) [0x55886f234106] 7: ceph-osd(+0x3e4d9c) [0x55886f225d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55886f7976c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55886f7a171d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55886f811061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55886f81468b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55886f815268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55886f802fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55886f803e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55886f70370d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55886f4c5592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55886f48d607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55886f483e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55886f3bfd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55886f609a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55886f3da727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55886f8d8d2a] 23: ceph-osd(+0xa982d4) [0x55886f8d92d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f479249f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f479243f450] 1 jobs: ['7626025'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/redirect}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x562cf3305fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f069e654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f069e6a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x562cf3305fa5] 6: ceph-osd(+0x3f3106) [0x562cf3306106] 7: ceph-osd(+0x3e4d9c) [0x562cf32f7d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x562cf38696c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x562cf387371d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x562cf38e3061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x562cf38e668b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x562cf38e7268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x562cf38d4fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x562cf38d5e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x562cf37d570d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x562cf3597592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x562cf355f607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x562cf3555e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x562cf3491d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562cf36dba19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x562cf34ac727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x562cf39aad2a] 23: ceph-osd(+0xa982d4) [0x562cf39ab2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f069e69f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f069e63f450] 1 jobs: ['7626095'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x557a4131bfa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f9b50454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f9b504a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x557a4131bfa5] 6: ceph-osd(+0x3f3106) [0x557a4131c106] 7: ceph-osd(+0x3e4d9c) [0x557a4130dd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x557a4187f6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x557a4188971d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x557a418f9061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x557a418fc68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x557a418fd268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x557a418eafe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x557a418ebe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x557a417eb70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x557a415ad592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x557a41575607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x557a4156be65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x557a414a7d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x557a416f1a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x557a414c2727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x557a419c0d2a] 23: ceph-osd(+0xa982d4) [0x557a419c12d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f9b5049f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f9b5043f450] 1 jobs: ['7626066'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41ed10b136a0de08323d8f65f00f42ebc2ccbcd8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 1 jobs: ['7626145'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_cephadm}'] Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi006 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41ed10b136a0de08323d8f65f00f42ebc2ccbcd8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 1 jobs: ['7626184'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_timeout}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560eed8b60bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4ee98e7520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560eed8b60bc] 6: ceph-osd(+0x3f721d) [0x560eed8b621d] 7: ceph-osd(+0x3e9888) [0x560eed8a8888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x560eede00c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x560eede1505e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x560eede762a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x560eede7f52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x560eede800a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x560eede84611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x560eede85433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x560eedd73e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x560eedd75937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x560eedb926f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x560eedaea74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x560eeda3eaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x560eedc83799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560eeda49793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x560eedf42313] 23: ceph-osd(+0xa83874) [0x560eedf42874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ee9939b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ee99cba00] 1 jobs: ['7626169'] suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/backfill-toofull', 'supported-random-distro$/{ubuntu_latest}}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5647fb40ffa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f26d6a54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f26d6aa154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5647fb40ffa5] 6: ceph-osd(+0x3f3106) [0x5647fb410106] 7: ceph-osd(+0x3e4d9c) [0x5647fb401d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5647fb9736c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5647fb97d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5647fb9ed061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5647fb9f068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5647fb9f1268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5647fb9defe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5647fb9dfe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5647fb8df70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5647fb6a1592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5647fb669607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5647fb65fe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5647fb59bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5647fb7e5a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5647fb5b6727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5647fbab4d2a] 23: ceph-osd(+0xa982d4) [0x5647fbab52d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f26d6a9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f26d6a3f450] 1 jobs: ['7626159'] suites: ['mon_election/classic', 'msgr-failures/none', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/singleton/{all/admin-socket', 'supported-random-distro$/{centos_latest}}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5566168a10bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f548682f520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5566168a10bc] 6: ceph-osd(+0x3f721d) [0x5566168a121d] 7: ceph-osd(+0x3e9888) [0x556616893888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x556616debc30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x556616e0005e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x556616e612a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x556616e6a52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x556616e6b0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x556616e6f611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x556616e70433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x556616d62c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x556616b18462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x556616aee96f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x556616af24e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x556616a29aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x556616c6e799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x556616a34793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x556616f2d313] 23: ceph-osd(+0xa83874) [0x556616f2d874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5486881b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5486913a00] 1 jobs: ['7626189'] suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_python}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5638141700bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f780c7f1520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5638141700bc] 6: ceph-osd(+0x3f721d) [0x56381417021d] 7: ceph-osd(+0x3e9888) [0x563814162888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5638146bac30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5638146cf05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5638147302a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56381473952b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56381473a0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x56381473e611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x56381473f433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x563814631c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5638143e7462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5638143bd96f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5638143c14e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5638142f8aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56381453d799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563814303793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5638147fc313] 23: ceph-osd(+0xa83874) [0x5638147fc874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f780c843b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f780c8d5a00] 1 jobs: ['7626054'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/write_fadvise_dontneed}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f5b5bf80d4a] 1 jobs: ['7626044'] suites: ['rados/objectstore/{backends/objectstore-bluestore-a', 'supported-random-distro$/{centos_latest}}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x558fa856a0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7941030520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x558fa856a0bc] 6: ceph-osd(+0x3f721d) [0x558fa856a21d] 7: ceph-osd(+0x3e9888) [0x558fa855c888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x558fa8ab4c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x558fa8ac905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x558fa8b2a2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x558fa8b3352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x558fa8b340a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x558fa8b38611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x558fa8b39433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x558fa8a27e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x558fa8a29937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x558fa88466f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x558fa879e74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x558fa86f2aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x558fa8937799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x558fa86fd793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x558fa8bf6313] 23: ceph-osd(+0xa83874) [0x558fa8bf6874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f7941082b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f7941114a00] 1 jobs: ['7626151'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache-snaps}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5642c2916fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f7931654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f79316a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5642c2916fa5] 6: ceph-osd(+0x3f3106) [0x5642c2917106] 7: ceph-osd(+0x3e4d9c) [0x5642c2908d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5642c2e7a6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5642c2e8471d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5642c2ef4061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5642c2ef768b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5642c2ef8268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5642c2ee5fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5642c2ee6e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5642c2de670d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5642c2ba8592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5642c2b70607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5642c2b66e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5642c2aa2d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5642c2ceca19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5642c2abd727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5642c2fbbd2a] 23: ceph-osd(+0xa982d4) [0x5642c2fbc2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f793169f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f793163f450] 1 jobs: ['7626069'] suites: ['mon_election/classic', 'msgr-failures/none', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5572eef45fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7feabe454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7feabe4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5572eef45fa5] 6: ceph-osd(+0x3f3106) [0x5572eef46106] 7: ceph-osd(+0x3e4d9c) [0x5572eef37d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5572ef4a96c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5572ef4b371d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5572ef523061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5572ef52668b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5572ef527268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5572ef514fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5572ef515e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5572ef41570d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5572ef1d7592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5572ef19f607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5572ef195e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5572ef0d1d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5572ef31ba19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5572ef0ec727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5572ef5ead2a] 23: ceph-osd(+0xa982d4) [0x5572ef5eb2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7feabe49f802] 25: /lib64/libc.so.6(+0x3f450) [0x7feabe43f450] 1 jobs: ['7626080'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55dee4436fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fce9d654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fce9d6a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55dee4436fa5] 6: ceph-osd(+0x3f3106) [0x55dee4437106] 7: ceph-osd(+0x3e4d9c) [0x55dee4428d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55dee499a6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55dee49a471d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55dee4a14061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55dee4a1768b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55dee4a18268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55dee4a05fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55dee4a06e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55dee490670d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55dee46c8592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55dee4690607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55dee4686e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55dee45c2d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55dee480ca19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55dee45dd727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55dee4adbd2a] 23: ceph-osd(+0xa982d4) [0x55dee4adc2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7fce9d69f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fce9d63f450] 1 jobs: ['7626052'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects}'] Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41ed10b136a0de08323d8f65f00f42ebc2ccbcd8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 1 jobs: ['7626180'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x556e37a2bfa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1d1ae54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f1d1aea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x556e37a2bfa5] 6: ceph-osd(+0x3f3106) [0x556e37a2c106] 7: ceph-osd(+0x3e4d9c) [0x556e37a1dd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x556e37f8f6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x556e37f9971d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x556e38009061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x556e3800c68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x556e3800d268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x556e37ffafe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x556e37ffbe33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x556e37f142a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x556e37ef9397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x556e37d33a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x556e37c7dedd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x556e37bb7d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x556e37e01a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x556e37bd2727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x556e380d0d2a] 23: ceph-osd(+0xa982d4) [0x556e380d12d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f1d1ae9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f1d1ae3f450] 1 jobs: ['7626051'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e55a7effa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7ff096654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7ff0966a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e55a7effa5] 6: ceph-osd(+0x3f3106) [0x55e55a7f0106] 7: ceph-osd(+0x3e4d9c) [0x55e55a7e1d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e55ad536c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55e55ad5d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55e55adcd061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e55add068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e55add1268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55e55adbefe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55e55adbfe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55e55acbf70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55e55aa81592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55e55aa49607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55e55aa3fe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e55a97bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e55abc5a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55e55a996727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55e55ae94d2a] 23: ceph-osd(+0xa982d4) [0x55e55ae952d4] 24: /lib64/libc.so.6(+0x9f802) [0x7ff09669f802] 25: /lib64/libc.so.6(+0x3f450) [0x7ff09663f450] 1 jobs: ['7626175'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/radosbench}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ba0aecdfa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8bc6e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8bc6ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ba0aecdfa5] 6: ceph-osd(+0x3f3106) [0x55ba0aece106] 7: ceph-osd(+0x3e4d9c) [0x55ba0aebfd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ba0b4316c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55ba0b43b71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55ba0b4ab061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ba0b4ae68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ba0b4af268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55ba0b49cfe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55ba0b49de33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x55ba0b3b62a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55ba0b39b397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55ba0b1d5a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x55ba0b11fedd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ba0b059d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ba0b2a3a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55ba0b074727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55ba0b572d2a] 23: ceph-osd(+0xa982d4) [0x55ba0b5732d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f8bc6e9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f8bc6e3f450] 1 jobs: ['7626013'] suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_latest}', 'thrashers/sync', 'workloads/rados_mon_workunits}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55d124cba0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0238cf4520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55d124cba0bc] 6: ceph-osd(+0x3f721d) [0x55d124cba21d] 7: ceph-osd(+0x3e9888) [0x55d124cac888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55d125204c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55d12521905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55d12527a2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55d12528352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55d1252840a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55d125288611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55d125289433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55d125177e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55d125179937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55d124f966f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55d124eee74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55d124e42aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d125087799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d124e4d793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55d125346313] 23: ceph-osd(+0xa83874) [0x55d125346874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0238d46b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f0238dd8a00] 1 jobs: ['7626201'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e51b197fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fb629454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fb6294a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e51b197fa5] 6: ceph-osd(+0x3f3106) [0x55e51b198106] 7: ceph-osd(+0x3e4d9c) [0x55e51b189d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e51b6fb6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55e51b70571d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55e51b775061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e51b77868b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e51b779268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55e51b766fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55e51b767e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55e51b66770d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55e51b429592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55e51b3f1607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55e51b3e7e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e51b323d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e51b56da19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55e51b33e727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55e51b83cd2a] 23: ceph-osd(+0xa982d4) [0x55e51b83d2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7fb62949f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fb62943f450] 1 jobs: ['7626097'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/radosbench}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x555a32dbd0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fbeef5de520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x555a32dbd0bc] 6: ceph-osd(+0x3f721d) [0x555a32dbd21d] 7: ceph-osd(+0x3e9888) [0x555a32daf888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x555a33307c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x555a3331c05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x555a3337d2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x555a3338652b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x555a333870a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x555a3338b611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x555a3338c433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x555a3327ec9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x555a33034462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x555a3300a96f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x555a3300e4e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x555a32f45aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x555a3318a799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x555a32f50793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x555a33449313] 23: ceph-osd(+0xa83874) [0x555a33449874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fbeef630b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fbeef6c2a00] 1 jobs: ['7626191'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/small-objects-localized}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x564c5f6b3fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f251a854db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f251a8a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x564c5f6b3fa5] 6: ceph-osd(+0x3f3106) [0x564c5f6b4106] 7: ceph-osd(+0x3e4d9c) [0x564c5f6a5d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x564c5fc176c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x564c5fc2171d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x564c5fc91061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x564c5fc9468b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x564c5fc95268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x564c5fc82fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x564c5fc83e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x564c5fb8370d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x564c5f945592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x564c5f90d607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x564c5f903e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x564c5f83fd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x564c5fa89a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x564c5f85a727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x564c5fd58d2a] 23: ceph-osd(+0xa982d4) [0x564c5fd592d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f251a89f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f251a83f450] 1 jobs: ['7626050'] suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton/{all/rebuild-mondb', 'supported-random-distro$/{centos_latest}}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55dc877670bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f00f2d5c520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55dc877670bc] 6: ceph-osd(+0x3f721d) [0x55dc8776721d] 7: ceph-osd(+0x3e9888) [0x55dc87759888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55dc87cb1c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55dc87cc605e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55dc87d272a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55dc87d3052b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55dc87d310a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55dc87d35611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55dc87d36433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55dc87c28c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55dc879de462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55dc879b496f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55dc879b84e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55dc878efaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55dc87b34799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55dc878fa793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55dc87df3313] 23: ceph-osd(+0xa83874) [0x55dc87df3874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f00f2daeb43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f00f2e40a00] 1 jobs: ['7626167'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/rados_api_tests}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ddb5e9ffa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f5176654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f51766a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ddb5e9ffa5] 6: ceph-osd(+0x3f3106) [0x55ddb5ea0106] 7: ceph-osd(+0x3e4d9c) [0x55ddb5e91d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ddb64036c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55ddb640d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55ddb647d061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ddb648068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ddb6481268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55ddb646efe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55ddb646fe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55ddb636f70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55ddb6131592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55ddb60f9607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55ddb60efe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ddb602bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ddb6275a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55ddb6046727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55ddb6544d2a] 23: ceph-osd(+0xa982d4) [0x55ddb65452d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f517669f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f517663f450] 1 jobs: ['7626197'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5652fc3240bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4bb8dde520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5652fc3240bc] 6: ceph-osd(+0x3f721d) [0x5652fc32421d] 7: ceph-osd(+0x3e9888) [0x5652fc316888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5652fc86ec30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5652fc88305e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5652fc8e42a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5652fc8ed52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5652fc8ee0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5652fc8f2611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5652fc8f3433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5652fc7e5c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5652fc59b462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5652fc57196f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5652fc5754e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5652fc4acaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5652fc6f1799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5652fc4b7793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5652fc9b0313] 23: ceph-osd(+0xa83874) [0x5652fc9b0874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4bb8e30b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4bb8ec2a00] 1 jobs: ['7626186'] suites: ['mon_election/connectivity', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/wpq_default_shards', 'settings/optimized', 'ubuntu_latest', 'workloads/fio_4K_rand_read}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x558da2496fa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f13db454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f13db4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x558da2496fa5] 6: ceph-osd(+0x3f3106) [0x558da2497106] 7: ceph-osd(+0x3e4d9c) [0x558da2488d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x558da29fa6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x558da2a0471d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x558da2a74061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x558da2a7768b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x558da2a78268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x558da2a65fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x558da2a66e33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x558da297f2a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x558da2964397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x558da279ea56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x558da26e8edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x558da2622d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x558da286ca19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x558da263d727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x558da2b3bd2a] 23: ceph-osd(+0xa982d4) [0x558da2b3c2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f13db49f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f13db43f450] 1 jobs: ['7626194'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/small-objects}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a1ed8cf0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1a0f925520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a1ed8cf0bc] 6: ceph-osd(+0x3f721d) [0x55a1ed8cf21d] 7: ceph-osd(+0x3e9888) [0x55a1ed8c1888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55a1ede19c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55a1ede2e05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55a1ede8f2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55a1ede9852b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55a1ede990a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55a1ede9d611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55a1ede9e433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55a1edd8ce3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55a1edd8e937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55a1edbab6f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55a1edb0374d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55a1eda57aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a1edc9c799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55a1eda62793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55a1edf5b313] 23: ceph-osd(+0xa83874) [0x55a1edf5b874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1a0f977b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f1a0fa09a00] 1 jobs: ['7626156'] suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_cls_all}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5557af04bfa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f7603054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f76030a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5557af04bfa5] 6: ceph-osd(+0x3f3106) [0x5557af04c106] 7: ceph-osd(+0x3e4d9c) [0x5557af03dd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5557af5af6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5557af5b971d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5557af629061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5557af62c68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5557af62d268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5557af61afe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5557af61be33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5557af51b70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5557af2dd592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5557af2a5607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5557af29be65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5557af1d7d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5557af421a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5557af1f2727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5557af6f0d2a] 23: ceph-osd(+0xa982d4) [0x5557af6f12d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f760309f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f760303f450] 1 jobs: ['7626057'] suites: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_latest}', 'thrashers/force-sync-many', 'workloads/snaps-few-objects}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c00b7330bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0ab3395520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c00b7330bc] 6: ceph-osd(+0x3f721d) [0x55c00b73321d] 7: ceph-osd(+0x3e9888) [0x55c00b725888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55c00bc7dc30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55c00bc9205e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55c00bcf32a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55c00bcfc52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55c00bcfd0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55c00bd01611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55c00bd02433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55c00bbf4c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55c00b9aa462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55c00b98096f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55c00b9844e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55c00b8bbaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c00bb00799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55c00b8c6793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55c00bdbf313] 23: ceph-osd(+0xa83874) [0x55c00bdbf874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0ab33e7b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f0ab3479a00] 1 jobs: ['7626124'] suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55fa7f73b0bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2b45958520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55fa7f73b0bc] 6: ceph-osd(+0x3f721d) [0x55fa7f73b21d] 7: ceph-osd(+0x3e9888) [0x55fa7f72d888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55fa7fc85c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55fa7fc9a05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55fa7fcfb2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55fa7fd0452b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55fa7fd050a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55fa7fd09611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55fa7fd0a433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55fa7fbf8e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55fa7fbfa937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55fa7fa176f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55fa7f96f74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55fa7f8c3aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fa7fb08799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55fa7f8ce793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55fa7fdc7313] 23: ceph-osd(+0xa83874) [0x55fa7fdc7874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2b459aab43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2b45a3ca00] 1 jobs: ['7626173'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5634a157ffa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f484c454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f484c4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5634a157ffa5] 6: ceph-osd(+0x3f3106) [0x5634a1580106] 7: ceph-osd(+0x3e4d9c) [0x5634a1571d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5634a1ae36c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5634a1aed71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5634a1b5d061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5634a1b6068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5634a1b61268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5634a1b4efe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5634a1b4fe33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x5634a1a682a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x5634a1a4d397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x5634a1887a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x5634a17d1edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5634a170bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5634a1955a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5634a1726727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5634a1c24d2a] 23: ceph-osd(+0xa982d4) [0x5634a1c252d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f484c49f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f484c43f450] 1 jobs: ['7626028'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/redirect_promote_tests}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e40ecdffa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f208f254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f208f2a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e40ecdffa5] 6: ceph-osd(+0x3f3106) [0x55e40ece0106] 7: ceph-osd(+0x3e4d9c) [0x55e40ecd1d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e40f2436c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55e40f24d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55e40f2bd061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e40f2c068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e40f2c1268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55e40f2aefe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55e40f2afe33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x55e40f1c82a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55e40f1ad397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55e40efe7a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x55e40ef31edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e40ee6bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e40f0b5a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55e40ee86727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55e40f384d2a] 23: ceph-osd(+0xa982d4) [0x55e40f3852d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f208f29f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f208f23f450] 1 jobs: ['7626063'] suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/scrub_test}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5634436a20bc] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7abceee520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5634436a20bc] 6: ceph-osd(+0x3f721d) [0x5634436a221d] 7: ceph-osd(+0x3e9888) [0x563443694888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x563443becc30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x563443c0105e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x563443c622a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x563443c6b52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x563443c6c0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x563443c70611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x563443c71433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x563443b63c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x563443919462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5634438ef96f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5634438f34e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56344382aaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563443a6f799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563443835793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x563443d2e313] 23: ceph-osd(+0xa83874) [0x563443d2e874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f7abcf40b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f7abcfd2a00] 1 jobs: ['7626196'] suites: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ff31f2dfa5] ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fb95e254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fb95e2a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ff31f2dfa5] 6: ceph-osd(+0x3f3106) [0x55ff31f2e106] 7: ceph-osd(+0x3e4d9c) [0x55ff31f1fd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ff324916c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55ff3249b71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55ff3250b061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ff3250e68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ff3250f268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55ff324fcfe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55ff324fde33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55ff323fd70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55ff321bf592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55ff32187607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55ff3217de65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ff320b9d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ff32303a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55ff320d4727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55ff325d2d2a] 23: ceph-osd(+0xa982d4) [0x55ff325d32d4] 24: /lib64/libc.so.6(+0x9f802) [0x7fb95e29f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fb95e23f450] 1 jobs: ['7626195'] suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-small-objects-many-deletes}'] Failure: reached maximum tries (51) after waiting for 300 seconds 1 jobs: ['7626085'] suites: ['2-inject-leak/osd', 'centos_latest}', 'rados/valgrind-leaks/{1-start'] Dead: None 1 jobs: ['7626065']