2024-04-01T09:48:55.200 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-01T09:48:55.200 INFO:teuthology.results:subset: '512/20000' 2024-04-01T09:48:55.200 INFO:teuthology.results:seed: '8345' 2024-04-01T11:41:04.947 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-01T11:41:04.978 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-01T23:17:07.111 INFO:teuthology.results:Tests finished! gathering results... 2024-04-01T23:17:22.718 INFO:scrape:Found 200 jobs 2024-04-01T23:17:33.726 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.726 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.739 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.742 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.754 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.754 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.765 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.766 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.777 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.778 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.789 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.789 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.801 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.801 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.813 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.813 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.825 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.825 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.836 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.836 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.849 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.849 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.861 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.861 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.873 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.874 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.885 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.885 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.897 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.897 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.910 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.910 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.921 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.921 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.933 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.933 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.949 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.949 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.961 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.962 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.973 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.973 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.985 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.985 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:33.997 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:33.997 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.009 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.009 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.020 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.020 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.031 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.032 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.048 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.048 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.060 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.060 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.072 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.072 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.083 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.084 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.095 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.095 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.107 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.108 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.122 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.122 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.135 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.135 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.147 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.147 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.162 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.162 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.175 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.175 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.187 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.188 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.205 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.205 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.221 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.222 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.234 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.235 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:34.246 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f5b5b654db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f5b5b6a154c] 2024-04-01T23:17:34.246 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-04-01T23:17:52.299 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.299 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.299 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.299 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.300 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.300 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.300 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.300 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.300 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.300 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.300 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.300 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.300 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.301 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.302 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.302 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.302 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.302 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.302 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.302 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.302 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.302 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.302 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.303 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.304 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.305 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.305 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.305 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.305 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.305 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/teuthology.log 2024-04-01T23:17:52.305 INFO:scrape:Found 66 distinct failure reasons 2024-04-01T23:17:52.306 INFO:scrape:Dead: b'2024-04-01T23:00:27.011 DEBUG:teuthology.exit:Finished running handlers' 2024-04-01T23:17:52.306 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7ff8b7054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7ff8b70a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55fd88e17fa5] 6: ceph-osd(+0x3f3106) [0x55fd88e18106] 7: ceph-osd(+0x3e4d9c) [0x55fd88e09d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55fd8937b6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55fd8938571d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55fd893f5061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55fd893f868b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55fd893f9268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55fd893e6fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55fd893e7e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55fd892e770d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55fd890a9592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55fd89071607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55fd89067e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55fd88fa3d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd891eda19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55fd88fbe727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55fd894bcd2a] 23: ceph-osd(+0xa982d4) [0x55fd894bd2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7ff8b709f802] 25: /lib64/libc.so.6(+0x3f450) [0x7ff8b703f450] 2024-04-01T23:17:52.306 INFO:scrape:3 jobs: ['7626119', '7626137', '7626048'] 2024-04-01T23:17:52.306 INFO:scrape:suites intersection: ['objectstore/bluestore-comp-lz4', 'rados'] 2024-04-01T23:17:52.306 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/none', 'thrashosds-health', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}'] 2024-04-01T23:17:52.306 INFO:scrape: 2024-04-01T23:17:52.306 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55bf1c4d2fa5] 2024-04-01T23:17:52.306 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f793d254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f793d2a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55bf1c4d2fa5] 6: ceph-osd(+0x3f3106) [0x55bf1c4d3106] 7: ceph-osd(+0x3e4d9c) [0x55bf1c4c4d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55bf1ca366c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55bf1ca4071d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55bf1cab0061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55bf1cab368b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55bf1cab4268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55bf1caa1fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55bf1caa2e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55bf1c9a270d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55bf1c764592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55bf1c72c607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55bf1c722e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55bf1c65ed27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55bf1c8a8a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55bf1c679727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55bf1cb77d2a] 23: ceph-osd(+0xa982d4) [0x55bf1cb782d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f793d29f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f793d23f450] 2024-04-01T23:17:52.306 INFO:scrape:1 jobs: ['7626148'] 2024-04-01T23:17:52.307 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-snaps-balanced}'] 2024-04-01T23:17:52.307 INFO:scrape: 2024-04-01T23:17:52.307 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x560b19ae6fa5] 2024-04-01T23:17:52.307 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1ab0054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f1ab00a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x560b19ae6fa5] 6: ceph-osd(+0x3f3106) [0x560b19ae7106] 7: ceph-osd(+0x3e4d9c) [0x560b19ad8d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x560b1a04a6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x560b1a05471d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x560b1a0c4061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x560b1a0c768b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x560b1a0c8268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x560b1a0b5fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x560b1a0b6e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x560b19fb670d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x560b19d78592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x560b19d40607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x560b19d36e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x560b19c72d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x560b19ebca19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x560b19c8d727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x560b1a18bd2a] 23: ceph-osd(+0xa982d4) [0x560b1a18c2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f1ab009f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f1ab003f450] 2024-04-01T23:17:52.307 INFO:scrape:1 jobs: ['7626130'] 2024-04-01T23:17:52.307 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects}'] 2024-04-01T23:17:52.307 INFO:scrape: 2024-04-01T23:17:52.307 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55f2faccefa5] 2024-04-01T23:17:52.308 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f2769454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f27694a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55f2faccefa5] 6: ceph-osd(+0x3f3106) [0x55f2faccf106] 7: ceph-osd(+0x3e4d9c) [0x55f2facc0d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55f2fb2326c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55f2fb23c71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55f2fb2ac061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55f2fb2af68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55f2fb2b0268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55f2fb29dfe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55f2fb29ee33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55f2fb19e70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55f2faf60592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55f2faf28607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55f2faf1ee65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55f2fae5ad27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f2fb0a4a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55f2fae75727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55f2fb373d2a] 23: ceph-osd(+0xa982d4) [0x55f2fb3742d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f276949f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f276943f450] 2024-04-01T23:17:52.308 INFO:scrape:1 jobs: ['7626075'] 2024-04-01T23:17:52.308 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-snaps}'] 2024-04-01T23:17:52.308 INFO:scrape: 2024-04-01T23:17:52.308 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-04-01T23:17:52.308 INFO:scrape:1 jobs: ['7626135'] 2024-04-01T23:17:52.308 INFO:scrape:suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}'] 2024-04-01T23:17:52.308 INFO:scrape: 2024-04-01T23:17:52.309 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x561c02dc4fa5] 2024-04-01T23:17:52.309 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fa94aa54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fa94aaa154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x561c02dc4fa5] 6: ceph-osd(+0x3f3106) [0x561c02dc5106] 7: ceph-osd(+0x3e4d9c) [0x561c02db6d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x561c033286c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x561c0333271d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x561c033a2061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x561c033a568b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x561c033a6268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x561c03393fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x561c03394e33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x561c032ad2a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x561c03292397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x561c030cca56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x561c03016edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x561c02f50d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561c0319aa19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x561c02f6b727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x561c03469d2a] 23: ceph-osd(+0xa982d4) [0x561c0346a2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7fa94aa9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fa94aa3f450] 2024-04-01T23:17:52.309 INFO:scrape:1 jobs: ['7626074'] 2024-04-01T23:17:52.309 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/cache-snaps-balanced}'] 2024-04-01T23:17:52.309 INFO:scrape: 2024-04-01T23:17:52.309 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e95f96ffa5] 2024-04-01T23:17:52.309 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8821e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8821ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e95f96ffa5] 6: ceph-osd(+0x3f3106) [0x55e95f970106] 7: ceph-osd(+0x3e4d9c) [0x55e95f961d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e95fed36c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55e95fedd71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55e95ff4d061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e95ff5068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e95ff51268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55e95ff3efe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55e95ff3fe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55e95fe3f70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55e95fc01592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55e95fbc9607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55e95fbbfe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e95fafbd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e95fd45a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55e95fb16727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55e960014d2a] 23: ceph-osd(+0xa982d4) [0x55e9600152d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f8821e9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f8821e3f450] 2024-04-01T23:17:52.309 INFO:scrape:1 jobs: ['7626070'] 2024-04-01T23:17:52.309 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/cache-pool-snaps}'] 2024-04-01T23:17:52.309 INFO:scrape: 2024-04-01T23:17:52.310 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5594cc35a0bc] 2024-04-01T23:17:52.310 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2d2c865520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5594cc35a0bc] 6: ceph-osd(+0x3f721d) [0x5594cc35a21d] 7: ceph-osd(+0x3e9888) [0x5594cc34c888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5594cc8a4c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5594cc8b905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5594cc91a2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5594cc92352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5594cc9240a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5594cc928611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5594cc929433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5594cc81bc9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5594cc5d1462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5594cc5a796f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5594cc5ab4e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5594cc4e2aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5594cc727799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5594cc4ed793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5594cc9e6313] 23: ceph-osd(+0xa83874) [0x5594cc9e6874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2d2c8b7b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2d2c949a00] 2024-04-01T23:17:52.310 INFO:scrape:1 jobs: ['7626150'] 2024-04-01T23:17:52.310 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2024-04-01T23:17:52.310 INFO:scrape: 2024-04-01T23:17:52.310 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x563420a4d0bc] 2024-04-01T23:17:52.310 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f27ce7e3520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x563420a4d0bc] 6: ceph-osd(+0x3f721d) [0x563420a4d21d] 7: ceph-osd(+0x3e9888) [0x563420a3f888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x563420f97c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x563420fac05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x56342100d2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56342101652b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5634210170a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x56342101b611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x56342101c433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x563420f0ae3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x563420f0c937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x563420d296f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x563420c8174d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x563420bd5aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563420e1a799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563420be0793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5634210d9313] 23: ceph-osd(+0xa83874) [0x5634210d9874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f27ce835b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f27ce8c7a00] 2024-04-01T23:17:52.310 INFO:scrape:1 jobs: ['7626121'] 2024-04-01T23:17:52.310 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] 2024-04-01T23:17:52.310 INFO:scrape: 2024-04-01T23:17:52.311 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55b355e280bc] 2024-04-01T23:17:52.311 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0426777520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55b355e280bc] 6: ceph-osd(+0x3f721d) [0x55b355e2821d] 7: ceph-osd(+0x3e9888) [0x55b355e1a888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55b356372c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55b35638705e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55b3563e82a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55b3563f152b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55b3563f20a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55b3563f6611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55b3563f7433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55b3562e9c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55b35609f462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55b35607596f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55b3560794e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55b355fb0aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b3561f5799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55b355fbb793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55b3564b4313] 23: ceph-osd(+0xa83874) [0x55b3564b4874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f04267c9b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f042685ba00] 2024-04-01T23:17:52.311 INFO:scrape:1 jobs: ['7626023'] 2024-04-01T23:17:52.311 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench}'] 2024-04-01T23:17:52.311 INFO:scrape: 2024-04-01T23:17:52.311 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55796bafffa5] 2024-04-01T23:17:52.311 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f277f454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f277f4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55796bafffa5] 6: ceph-osd(+0x3f3106) [0x55796bb00106] 7: ceph-osd(+0x3e4d9c) [0x55796baf1d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55796c0636c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55796c06d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55796c0dd061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55796c0e068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55796c0e1268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55796c0cefe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55796c0cfe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55796bfcf70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55796bd91592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55796bd59607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55796bd4fe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55796bc8bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55796bed5a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55796bca6727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55796c1a4d2a] 23: ceph-osd(+0xa982d4) [0x55796c1a52d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f277f49f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f277f43f450] 2024-04-01T23:17:52.311 INFO:scrape:1 jobs: ['7626129'] 2024-04-01T23:17:52.311 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/rados_api_tests}'] 2024-04-01T23:17:52.311 INFO:scrape: 2024-04-01T23:17:52.312 INFO:scrape:Failure: "2024-04-01T11:14:54.456109+0000 mon.smithi080 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log 2024-04-01T23:17:52.312 INFO:scrape:8 jobs: ['7626125', '7626152', '7626072', '7626093', '7626177', '7626199', '7626042', '7626012'] 2024-04-01T23:17:52.312 INFO:scrape:suites intersection: [] 2024-04-01T23:17:52.312 INFO:scrape:suites union: ['0-nvme-loop', '0-start', '1-start', '1-tasks', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', 'mon_election/connectivity', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-04-01T23:17:52.312 INFO:scrape: 2024-04-01T23:17:52.312 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5597371c00bc] 2024-04-01T23:17:52.312 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0badbb9520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5597371c00bc] 6: ceph-osd(+0x3f721d) [0x5597371c021d] 7: ceph-osd(+0x3e9888) [0x5597371b2888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55973770ac30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55973771f05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5597377802a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55973778952b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55973778a0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55973778e611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55973778f433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55973767de3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55973767f937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55973749c6f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x5597373f474d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x559737348aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55973758d799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x559737353793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55973784c313] 23: ceph-osd(+0xa83874) [0x55973784c874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0badc0bb43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f0badc9da00] 2024-04-01T23:17:52.313 INFO:scrape:1 jobs: ['7626100'] 2024-04-01T23:17:52.313 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/redirect}'] 2024-04-01T23:17:52.313 INFO:scrape: 2024-04-01T23:17:52.313 INFO:scrape:Dead: b'2024-04-01T21:56:21.199 DEBUG:teuthology.exit:Finished running handlers' 2024-04-01T23:17:52.313 INFO:scrape:1 jobs: ['7626010'] 2024-04-01T23:17:52.313 INFO:scrape:suites: ['mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-01T23:17:52.313 INFO:scrape: 2024-04-01T23:17:52.313 INFO:scrape:Failure: No module named 'tasks.ceph' 2024-04-01T23:17:52.313 INFO:scrape:1 jobs: ['7626204'] 2024-04-01T23:17:52.313 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2024-04-01T23:17:52.314 INFO:scrape: 2024-04-01T23:17:52.314 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ccd46a70bc] 2024-04-01T23:17:52.314 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8ef20eb520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ccd46a70bc] 6: ceph-osd(+0x3f721d) [0x55ccd46a721d] 7: ceph-osd(+0x3e9888) [0x55ccd4699888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ccd4bf1c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55ccd4c0605e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55ccd4c672a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ccd4c7052b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ccd4c710a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55ccd4c75611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55ccd4c76433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55ccd4b64e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55ccd4b66937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55ccd49836f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55ccd48db74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ccd482faa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ccd4a74799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55ccd483a793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55ccd4d33313] 23: ceph-osd(+0xa83874) [0x55ccd4d33874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f8ef213db43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f8ef21cfa00] 2024-04-01T23:17:52.314 INFO:scrape:1 jobs: ['7626046'] 2024-04-01T23:17:52.314 INFO:scrape:suites: ['mon_election/classic', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/dmclock_1Shard_16Threads', 'settings/optimized', 'ubuntu_latest', 'workloads/radosbench_4M_seq_read}'] 2024-04-01T23:17:52.314 INFO:scrape: 2024-04-01T23:17:52.314 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5621760280bc] 2024-04-01T23:17:52.314 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f976bd45520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5621760280bc] 6: ceph-osd(+0x3f721d) [0x56217602821d] 7: ceph-osd(+0x3e9888) [0x56217601a888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x562176572c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x56217658705e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5621765e82a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5621765f152b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5621765f20a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5621765f6611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5621765f7433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5621764e9c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x56217629f462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x56217627596f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5621762794e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5621761b0aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5621763f5799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5621761bb793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5621766b4313] 23: ceph-osd(+0xa83874) [0x5621766b4874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f976bd97b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f976be29a00] 2024-04-01T23:17:52.314 INFO:scrape:1 jobs: ['7626126'] 2024-04-01T23:17:52.314 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] 2024-04-01T23:17:52.314 INFO:scrape: 2024-04-01T23:17:52.315 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x559b630affa5] 2024-04-01T23:17:52.315 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8cf7454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8cf74a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x559b630affa5] 6: ceph-osd(+0x3f3106) [0x559b630b0106] 7: ceph-osd(+0x3e4d9c) [0x559b630a1d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x559b636136c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x559b6361d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x559b6368d061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x559b6369068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x559b63691268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x559b6367efe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x559b6367fe33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x559b635982a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x559b6357d397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x559b633b7a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x559b63301edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x559b6323bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x559b63485a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x559b63256727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x559b63754d2a] 23: ceph-osd(+0xa982d4) [0x559b637552d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f8cf749f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f8cf743f450] 2024-04-01T23:17:52.315 INFO:scrape:1 jobs: ['7626032'] 2024-04-01T23:17:52.315 INFO:scrape:suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}'] 2024-04-01T23:17:52.315 INFO:scrape: 2024-04-01T23:17:52.315 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55732de3c0bc] 2024-04-01T23:17:52.315 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7feb548af520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55732de3c0bc] 6: ceph-osd(+0x3f721d) [0x55732de3c21d] 7: ceph-osd(+0x3e9888) [0x55732de2e888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55732e386c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55732e39b05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55732e3fc2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55732e40552b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55732e4060a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55732e40a611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55732e40b433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55732e2fdc9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55732e0b3462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55732e08996f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55732e08d4e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55732dfc4aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55732e209799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55732dfcf793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55732e4c8313] 23: ceph-osd(+0xa83874) [0x55732e4c8874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7feb54901b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7feb54993a00] 2024-04-01T23:17:52.315 INFO:scrape:1 jobs: ['7626176'] 2024-04-01T23:17:52.315 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/redirect}'] 2024-04-01T23:17:52.315 INFO:scrape: 2024-04-01T23:17:52.316 INFO:scrape:Failure: Command failed on smithi070 with status 5: 'sudo systemctl stop ceph-966734f0-f01a-11ee-b647-cb9ed24678a4@mon.a' 2024-04-01T23:17:52.316 INFO:scrape:1 jobs: ['7626140'] 2024-04-01T23:17:52.316 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] 2024-04-01T23:17:52.316 INFO:scrape: 2024-04-01T23:17:52.316 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55888501d0bc] 2024-04-01T23:17:52.316 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ffa87b9e520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55888501d0bc] 6: ceph-osd(+0x3f721d) [0x55888501d21d] 7: ceph-osd(+0x3e9888) [0x55888500f888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x558885567c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55888557c05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5588855dd2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5588855e652b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5588855e70a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5588855eb611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5588855ec433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x5588854dae3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x5588854dc937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x5588852f96f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55888525174d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5588851a5aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5588853ea799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5588851b0793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5588856a9313] 23: ceph-osd(+0xa83874) [0x5588856a9874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ffa87bf0b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ffa87c82a00] 2024-04-01T23:17:52.316 INFO:scrape:1 jobs: ['7626105'] 2024-04-01T23:17:52.316 INFO:scrape:suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/many', 'workloads/pool-create-delete}'] 2024-04-01T23:17:52.316 INFO:scrape: 2024-04-01T23:17:52.316 INFO:scrape:Assertion: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x4faf4a] 2024-04-01T23:17:52.316 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x4faf4a] 2: ceph-osd(+0x3f3106) [0x4fb106] 3: ceph-osd(+0x3e4d9c) [0x4ecd9c] 4: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0xa5e6c0] 5: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0xa6871d] 6: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0xad8061] 7: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0xadb68b] 8: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0xadc268] 9: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0xac9fe5] 10: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0xacae33] 11: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x9ca70d] 12: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x78c592] 13: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x754607] 14: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x74ae65] 15: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x686d27] 16: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8d0a19] 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6a1727] 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xb9fd2a] 19: ceph-osd(+0xa982d4) [0xba02d4] 20: /lib64/libc.so.6(+0x9f802) [0x54eb802] 2024-04-01T23:17:52.317 INFO:scrape:1 jobs: ['7626021'] 2024-04-01T23:17:52.317 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'validater/valgrind}'] 2024-04-01T23:17:52.317 INFO:scrape: 2024-04-01T23:17:52.317 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56426a43cfa5] 2024-04-01T23:17:52.317 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f7557054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f75570a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56426a43cfa5] 6: ceph-osd(+0x3f3106) [0x56426a43d106] 7: ceph-osd(+0x3e4d9c) [0x56426a42ed9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x56426a9a06c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56426a9aa71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x56426aa1a061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56426aa1d68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56426aa1e268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x56426aa0bfe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x56426aa0ce33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x56426a90c70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x56426a6ce592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x56426a696607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x56426a68ce65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56426a5c8d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56426a812a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56426a5e3727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56426aae1d2a] 23: ceph-osd(+0xa982d4) [0x56426aae22d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f755709f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f755703f450] 2024-04-01T23:17:52.317 INFO:scrape:1 jobs: ['7626077'] 2024-04-01T23:17:52.317 INFO:scrape:suites: ['cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] 2024-04-01T23:17:52.317 INFO:scrape: 2024-04-01T23:17:52.317 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55db39737fa5] 2024-04-01T23:17:52.317 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f0896e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f0896ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55db39737fa5] 6: ceph-osd(+0x3f3106) [0x55db39738106] 7: ceph-osd(+0x3e4d9c) [0x55db39729d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55db39c9b6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55db39ca571d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55db39d15061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55db39d1868b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55db39d19268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55db39d06fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55db39d07e33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x55db39c202a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55db39c05397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55db39a3fa56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x55db39989edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55db398c3d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55db39b0da19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55db398de727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55db39ddcd2a] 23: ceph-osd(+0xa982d4) [0x55db39ddd2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f0896e9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f0896e3f450] 2024-04-01T23:17:52.318 INFO:scrape:1 jobs: ['7626029'] 2024-04-01T23:17:52.318 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/repair_test}'] 2024-04-01T23:17:52.318 INFO:scrape: 2024-04-01T23:17:52.318 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5639750a3fa5] 2024-04-01T23:17:52.318 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f685d054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f685d0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5639750a3fa5] 6: ceph-osd(+0x3f3106) [0x5639750a4106] 7: ceph-osd(+0x3e4d9c) [0x563975095d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5639756076c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56397561171d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x563975681061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56397568468b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x563975685268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x563975672fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x563975673e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x56397557370d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x563975335592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5639752fd607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5639752f3e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56397522fd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563975479a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56397524a727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x563975748d2a] 23: ceph-osd(+0xa982d4) [0x5639757492d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f685d09f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f685d03f450] 2024-04-01T23:17:52.318 INFO:scrape:1 jobs: ['7626106'] 2024-04-01T23:17:52.318 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/redirect_promote_tests}'] 2024-04-01T23:17:52.318 INFO:scrape: 2024-04-01T23:17:52.318 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56413f9c5fa5] 2024-04-01T23:17:52.318 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1aa6454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f1aa64a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56413f9c5fa5] 6: ceph-osd(+0x3f3106) [0x56413f9c6106] 7: ceph-osd(+0x3e4d9c) [0x56413f9b7d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x56413ff296c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56413ff3371d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x56413ffa3061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56413ffa668b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56413ffa7268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x56413ff94fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x56413ff95e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x56413fe9570d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x56413fc57592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x56413fc1f607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x56413fc15e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56413fb51d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56413fd9ba19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56413fb6c727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56414006ad2a] 23: ceph-osd(+0xa982d4) [0x56414006b2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f1aa649f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f1aa643f450] 2024-04-01T23:17:52.319 INFO:scrape:1 jobs: ['7626144'] 2024-04-01T23:17:52.319 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/cache-pool-snaps}'] 2024-04-01T23:17:52.319 INFO:scrape: 2024-04-01T23:17:52.319 INFO:scrape:Failure: wait_for_clean: failed before timeout expired 2024-04-01T23:17:52.319 INFO:scrape:2 jobs: ['7626082', '7626160'] 2024-04-01T23:17:52.319 INFO:scrape:suites intersection: ['debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}'] 2024-04-01T23:17:52.319 INFO:scrape:suites union: ['debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-snappy}', 'random-objectstore$/{bluestore-comp-zlib}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/insights}', 'tasks/per_module_finisher_stats}'] 2024-04-01T23:17:52.319 INFO:scrape: 2024-04-01T23:17:52.319 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55956db7a0bc] 2024-04-01T23:17:52.319 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f04560b6520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55956db7a0bc] 6: ceph-osd(+0x3f721d) [0x55956db7a21d] 7: ceph-osd(+0x3e9888) [0x55956db6c888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55956e0c4c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55956e0d905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55956e13a2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55956e14352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55956e1440a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55956e148611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55956e149433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55956e03bc9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55956ddf1462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55956ddc796f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55956ddcb4e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55956dd02aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55956df47799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55956dd0d793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55956e206313] 23: ceph-osd(+0xa83874) [0x55956e206874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0456108b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f045619aa00] 2024-04-01T23:17:52.319 INFO:scrape:1 jobs: ['7626033'] 2024-04-01T23:17:52.320 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/redirect_set_object}'] 2024-04-01T23:17:52.320 INFO:scrape: 2024-04-01T23:17:52.320 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560af16fa0bc] 2024-04-01T23:17:52.320 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5e32d1e520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560af16fa0bc] 6: ceph-osd(+0x3f721d) [0x560af16fa21d] 7: ceph-osd(+0x3e9888) [0x560af16ec888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x560af1c44c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x560af1c5905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x560af1cba2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x560af1cc352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x560af1cc40a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x560af1cc8611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x560af1cc9433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x560af1bb7e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x560af1bb9937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x560af19d66f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x560af192e74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x560af1882aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x560af1ac7799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560af188d793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x560af1d86313] 23: ceph-osd(+0xa83874) [0x560af1d86874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5e32d70b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5e32e02a00] 2024-04-01T23:17:52.320 INFO:scrape:1 jobs: ['7626142'] 2024-04-01T23:17:52.320 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}'] 2024-04-01T23:17:52.320 INFO:scrape: 2024-04-01T23:17:52.320 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-01T23:17:52.320 INFO:scrape:1 jobs: ['7626112'] 2024-04-01T23:17:52.320 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-04-01T23:17:52.320 INFO:scrape: 2024-04-01T23:17:52.321 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55886f233fa5] 2024-04-01T23:17:52.321 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f4792454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f47924a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55886f233fa5] 6: ceph-osd(+0x3f3106) [0x55886f234106] 7: ceph-osd(+0x3e4d9c) [0x55886f225d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55886f7976c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55886f7a171d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55886f811061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55886f81468b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55886f815268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55886f802fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55886f803e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55886f70370d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55886f4c5592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55886f48d607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55886f483e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55886f3bfd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55886f609a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55886f3da727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55886f8d8d2a] 23: ceph-osd(+0xa982d4) [0x55886f8d92d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f479249f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f479243f450] 2024-04-01T23:17:52.321 INFO:scrape:1 jobs: ['7626025'] 2024-04-01T23:17:52.321 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/redirect}'] 2024-04-01T23:17:52.321 INFO:scrape: 2024-04-01T23:17:52.321 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x562cf3305fa5] 2024-04-01T23:17:52.321 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f069e654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f069e6a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x562cf3305fa5] 6: ceph-osd(+0x3f3106) [0x562cf3306106] 7: ceph-osd(+0x3e4d9c) [0x562cf32f7d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x562cf38696c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x562cf387371d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x562cf38e3061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x562cf38e668b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x562cf38e7268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x562cf38d4fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x562cf38d5e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x562cf37d570d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x562cf3597592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x562cf355f607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x562cf3555e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x562cf3491d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562cf36dba19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x562cf34ac727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x562cf39aad2a] 23: ceph-osd(+0xa982d4) [0x562cf39ab2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f069e69f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f069e63f450] 2024-04-01T23:17:52.321 INFO:scrape:1 jobs: ['7626095'] 2024-04-01T23:17:52.321 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] 2024-04-01T23:17:52.321 INFO:scrape: 2024-04-01T23:17:52.321 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x557a4131bfa5] 2024-04-01T23:17:52.322 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f9b50454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f9b504a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x557a4131bfa5] 6: ceph-osd(+0x3f3106) [0x557a4131c106] 7: ceph-osd(+0x3e4d9c) [0x557a4130dd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x557a4187f6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x557a4188971d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x557a418f9061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x557a418fc68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x557a418fd268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x557a418eafe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x557a418ebe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x557a417eb70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x557a415ad592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x557a41575607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x557a4156be65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x557a414a7d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x557a416f1a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x557a414c2727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x557a419c0d2a] 23: ceph-osd(+0xa982d4) [0x557a419c12d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f9b5049f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f9b5043f450] 2024-04-01T23:17:52.322 INFO:scrape:1 jobs: ['7626066'] 2024-04-01T23:17:52.322 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] 2024-04-01T23:17:52.322 INFO:scrape: 2024-04-01T23:17:52.322 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41ed10b136a0de08323d8f65f00f42ebc2ccbcd8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-04-01T23:17:52.322 INFO:scrape:1 jobs: ['7626145'] 2024-04-01T23:17:52.322 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_cephadm}'] 2024-04-01T23:17:52.322 INFO:scrape: 2024-04-01T23:17:52.322 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi006 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41ed10b136a0de08323d8f65f00f42ebc2ccbcd8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 2024-04-01T23:17:52.322 INFO:scrape:1 jobs: ['7626184'] 2024-04-01T23:17:52.323 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_timeout}'] 2024-04-01T23:17:52.323 INFO:scrape: 2024-04-01T23:17:52.323 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560eed8b60bc] 2024-04-01T23:17:52.323 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4ee98e7520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560eed8b60bc] 6: ceph-osd(+0x3f721d) [0x560eed8b621d] 7: ceph-osd(+0x3e9888) [0x560eed8a8888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x560eede00c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x560eede1505e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x560eede762a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x560eede7f52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x560eede800a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x560eede84611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x560eede85433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x560eedd73e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x560eedd75937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x560eedb926f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x560eedaea74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x560eeda3eaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x560eedc83799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560eeda49793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x560eedf42313] 23: ceph-osd(+0xa83874) [0x560eedf42874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ee9939b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ee99cba00] 2024-04-01T23:17:52.323 INFO:scrape:1 jobs: ['7626169'] 2024-04-01T23:17:52.323 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/backfill-toofull', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-01T23:17:52.323 INFO:scrape: 2024-04-01T23:17:52.323 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5647fb40ffa5] 2024-04-01T23:17:52.323 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f26d6a54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f26d6aa154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5647fb40ffa5] 6: ceph-osd(+0x3f3106) [0x5647fb410106] 7: ceph-osd(+0x3e4d9c) [0x5647fb401d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5647fb9736c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5647fb97d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5647fb9ed061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5647fb9f068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5647fb9f1268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5647fb9defe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5647fb9dfe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5647fb8df70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5647fb6a1592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5647fb669607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5647fb65fe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5647fb59bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5647fb7e5a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5647fb5b6727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5647fbab4d2a] 23: ceph-osd(+0xa982d4) [0x5647fbab52d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f26d6a9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f26d6a3f450] 2024-04-01T23:17:52.323 INFO:scrape:1 jobs: ['7626159'] 2024-04-01T23:17:52.323 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/none', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/singleton/{all/admin-socket', 'supported-random-distro$/{centos_latest}}'] 2024-04-01T23:17:52.324 INFO:scrape: 2024-04-01T23:17:52.324 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5566168a10bc] 2024-04-01T23:17:52.324 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f548682f520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5566168a10bc] 6: ceph-osd(+0x3f721d) [0x5566168a121d] 7: ceph-osd(+0x3e9888) [0x556616893888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x556616debc30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x556616e0005e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x556616e612a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x556616e6a52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x556616e6b0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x556616e6f611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x556616e70433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x556616d62c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x556616b18462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x556616aee96f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x556616af24e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x556616a29aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x556616c6e799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x556616a34793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x556616f2d313] 23: ceph-osd(+0xa83874) [0x556616f2d874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5486881b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5486913a00] 2024-04-01T23:17:52.324 INFO:scrape:1 jobs: ['7626189'] 2024-04-01T23:17:52.324 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_python}'] 2024-04-01T23:17:52.324 INFO:scrape: 2024-04-01T23:17:52.324 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5638141700bc] 2024-04-01T23:17:52.324 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f780c7f1520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5638141700bc] 6: ceph-osd(+0x3f721d) [0x56381417021d] 7: ceph-osd(+0x3e9888) [0x563814162888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5638146bac30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5638146cf05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5638147302a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56381473952b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56381473a0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x56381473e611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x56381473f433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x563814631c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5638143e7462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5638143bd96f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5638143c14e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5638142f8aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56381453d799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563814303793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5638147fc313] 23: ceph-osd(+0xa83874) [0x5638147fc874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f780c843b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f780c8d5a00] 2024-04-01T23:17:52.324 INFO:scrape:1 jobs: ['7626054'] 2024-04-01T23:17:52.324 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/write_fadvise_dontneed}'] 2024-04-01T23:17:52.324 INFO:scrape: 2024-04-01T23:17:52.325 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f5b5bf80d4a] 2024-04-01T23:17:52.325 INFO:scrape:1 jobs: ['7626044'] 2024-04-01T23:17:52.325 INFO:scrape:suites: ['rados/objectstore/{backends/objectstore-bluestore-a', 'supported-random-distro$/{centos_latest}}'] 2024-04-01T23:17:52.325 INFO:scrape: 2024-04-01T23:17:52.325 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x558fa856a0bc] 2024-04-01T23:17:52.325 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7941030520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x558fa856a0bc] 6: ceph-osd(+0x3f721d) [0x558fa856a21d] 7: ceph-osd(+0x3e9888) [0x558fa855c888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x558fa8ab4c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x558fa8ac905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x558fa8b2a2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x558fa8b3352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x558fa8b340a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x558fa8b38611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x558fa8b39433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x558fa8a27e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x558fa8a29937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x558fa88466f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x558fa879e74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x558fa86f2aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x558fa8937799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x558fa86fd793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x558fa8bf6313] 23: ceph-osd(+0xa83874) [0x558fa8bf6874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f7941082b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f7941114a00] 2024-04-01T23:17:52.325 INFO:scrape:1 jobs: ['7626151'] 2024-04-01T23:17:52.325 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache-snaps}'] 2024-04-01T23:17:52.325 INFO:scrape: 2024-04-01T23:17:52.325 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5642c2916fa5] 2024-04-01T23:17:52.325 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f7931654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f79316a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5642c2916fa5] 6: ceph-osd(+0x3f3106) [0x5642c2917106] 7: ceph-osd(+0x3e4d9c) [0x5642c2908d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5642c2e7a6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5642c2e8471d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5642c2ef4061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5642c2ef768b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5642c2ef8268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5642c2ee5fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5642c2ee6e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5642c2de670d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5642c2ba8592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5642c2b70607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5642c2b66e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5642c2aa2d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5642c2ceca19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5642c2abd727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5642c2fbbd2a] 23: ceph-osd(+0xa982d4) [0x5642c2fbc2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f793169f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f793163f450] 2024-04-01T23:17:52.326 INFO:scrape:1 jobs: ['7626069'] 2024-04-01T23:17:52.326 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/none', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] 2024-04-01T23:17:52.326 INFO:scrape: 2024-04-01T23:17:52.326 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5572eef45fa5] 2024-04-01T23:17:52.326 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7feabe454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7feabe4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5572eef45fa5] 6: ceph-osd(+0x3f3106) [0x5572eef46106] 7: ceph-osd(+0x3e4d9c) [0x5572eef37d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5572ef4a96c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5572ef4b371d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5572ef523061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5572ef52668b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5572ef527268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5572ef514fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5572ef515e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5572ef41570d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5572ef1d7592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5572ef19f607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5572ef195e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5572ef0d1d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5572ef31ba19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5572ef0ec727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5572ef5ead2a] 23: ceph-osd(+0xa982d4) [0x5572ef5eb2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7feabe49f802] 25: /lib64/libc.so.6(+0x3f450) [0x7feabe43f450] 2024-04-01T23:17:52.326 INFO:scrape:1 jobs: ['7626080'] 2024-04-01T23:17:52.326 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache}'] 2024-04-01T23:17:52.326 INFO:scrape: 2024-04-01T23:17:52.327 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55dee4436fa5] 2024-04-01T23:17:52.327 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fce9d654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fce9d6a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55dee4436fa5] 6: ceph-osd(+0x3f3106) [0x55dee4437106] 7: ceph-osd(+0x3e4d9c) [0x55dee4428d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55dee499a6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55dee49a471d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55dee4a14061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55dee4a1768b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55dee4a18268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55dee4a05fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55dee4a06e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55dee490670d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55dee46c8592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55dee4690607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55dee4686e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55dee45c2d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55dee480ca19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55dee45dd727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55dee4adbd2a] 23: ceph-osd(+0xa982d4) [0x55dee4adc2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7fce9d69f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fce9d63f450] 2024-04-01T23:17:52.327 INFO:scrape:1 jobs: ['7626052'] 2024-04-01T23:17:52.327 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects}'] 2024-04-01T23:17:52.327 INFO:scrape: 2024-04-01T23:17:52.327 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41ed10b136a0de08323d8f65f00f42ebc2ccbcd8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-04-01T23:17:52.327 INFO:scrape:1 jobs: ['7626180'] 2024-04-01T23:17:52.327 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] 2024-04-01T23:17:52.327 INFO:scrape: 2024-04-01T23:17:52.327 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x556e37a2bfa5] 2024-04-01T23:17:52.327 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1d1ae54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f1d1aea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x556e37a2bfa5] 6: ceph-osd(+0x3f3106) [0x556e37a2c106] 7: ceph-osd(+0x3e4d9c) [0x556e37a1dd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x556e37f8f6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x556e37f9971d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x556e38009061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x556e3800c68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x556e3800d268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x556e37ffafe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x556e37ffbe33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x556e37f142a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x556e37ef9397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x556e37d33a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x556e37c7dedd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x556e37bb7d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x556e37e01a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x556e37bd2727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x556e380d0d2a] 23: ceph-osd(+0xa982d4) [0x556e380d12d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f1d1ae9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f1d1ae3f450] 2024-04-01T23:17:52.328 INFO:scrape:1 jobs: ['7626051'] 2024-04-01T23:17:52.329 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] 2024-04-01T23:17:52.329 INFO:scrape: 2024-04-01T23:17:52.329 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e55a7effa5] 2024-04-01T23:17:52.329 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7ff096654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7ff0966a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e55a7effa5] 6: ceph-osd(+0x3f3106) [0x55e55a7f0106] 7: ceph-osd(+0x3e4d9c) [0x55e55a7e1d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e55ad536c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55e55ad5d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55e55adcd061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e55add068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e55add1268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55e55adbefe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55e55adbfe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55e55acbf70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55e55aa81592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55e55aa49607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55e55aa3fe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e55a97bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e55abc5a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55e55a996727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55e55ae94d2a] 23: ceph-osd(+0xa982d4) [0x55e55ae952d4] 24: /lib64/libc.so.6(+0x9f802) [0x7ff09669f802] 25: /lib64/libc.so.6(+0x3f450) [0x7ff09663f450] 2024-04-01T23:17:52.329 INFO:scrape:1 jobs: ['7626175'] 2024-04-01T23:17:52.329 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/radosbench}'] 2024-04-01T23:17:52.329 INFO:scrape: 2024-04-01T23:17:52.329 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ba0aecdfa5] 2024-04-01T23:17:52.329 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8bc6e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8bc6ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ba0aecdfa5] 6: ceph-osd(+0x3f3106) [0x55ba0aece106] 7: ceph-osd(+0x3e4d9c) [0x55ba0aebfd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ba0b4316c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55ba0b43b71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55ba0b4ab061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ba0b4ae68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ba0b4af268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55ba0b49cfe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55ba0b49de33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x55ba0b3b62a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55ba0b39b397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55ba0b1d5a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x55ba0b11fedd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ba0b059d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ba0b2a3a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55ba0b074727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55ba0b572d2a] 23: ceph-osd(+0xa982d4) [0x55ba0b5732d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f8bc6e9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f8bc6e3f450] 2024-04-01T23:17:52.330 INFO:scrape:1 jobs: ['7626013'] 2024-04-01T23:17:52.330 INFO:scrape:suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_latest}', 'thrashers/sync', 'workloads/rados_mon_workunits}'] 2024-04-01T23:17:52.330 INFO:scrape: 2024-04-01T23:17:52.330 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55d124cba0bc] 2024-04-01T23:17:52.330 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0238cf4520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55d124cba0bc] 6: ceph-osd(+0x3f721d) [0x55d124cba21d] 7: ceph-osd(+0x3e9888) [0x55d124cac888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55d125204c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55d12521905e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55d12527a2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55d12528352b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55d1252840a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55d125288611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55d125289433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55d125177e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55d125179937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55d124f966f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55d124eee74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55d124e42aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d125087799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d124e4d793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55d125346313] 23: ceph-osd(+0xa83874) [0x55d125346874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0238d46b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f0238dd8a00] 2024-04-01T23:17:52.330 INFO:scrape:1 jobs: ['7626201'] 2024-04-01T23:17:52.330 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/read', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] 2024-04-01T23:17:52.330 INFO:scrape: 2024-04-01T23:17:52.330 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e51b197fa5] 2024-04-01T23:17:52.330 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fb629454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fb6294a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e51b197fa5] 6: ceph-osd(+0x3f3106) [0x55e51b198106] 7: ceph-osd(+0x3e4d9c) [0x55e51b189d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e51b6fb6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55e51b70571d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55e51b775061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e51b77868b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e51b779268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55e51b766fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55e51b767e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55e51b66770d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55e51b429592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55e51b3f1607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55e51b3e7e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e51b323d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e51b56da19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55e51b33e727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55e51b83cd2a] 23: ceph-osd(+0xa982d4) [0x55e51b83d2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7fb62949f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fb62943f450] 2024-04-01T23:17:52.331 INFO:scrape:1 jobs: ['7626097'] 2024-04-01T23:17:52.331 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/radosbench}'] 2024-04-01T23:17:52.331 INFO:scrape: 2024-04-01T23:17:52.331 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x555a32dbd0bc] 2024-04-01T23:17:52.331 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fbeef5de520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x555a32dbd0bc] 6: ceph-osd(+0x3f721d) [0x555a32dbd21d] 7: ceph-osd(+0x3e9888) [0x555a32daf888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x555a33307c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x555a3331c05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x555a3337d2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x555a3338652b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x555a333870a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x555a3338b611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x555a3338c433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x555a3327ec9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x555a33034462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x555a3300a96f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x555a3300e4e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x555a32f45aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x555a3318a799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x555a32f50793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x555a33449313] 23: ceph-osd(+0xa83874) [0x555a33449874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fbeef630b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fbeef6c2a00] 2024-04-01T23:17:52.331 INFO:scrape:1 jobs: ['7626191'] 2024-04-01T23:17:52.331 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/small-objects-localized}'] 2024-04-01T23:17:52.331 INFO:scrape: 2024-04-01T23:17:52.331 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x564c5f6b3fa5] 2024-04-01T23:17:52.331 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f251a854db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f251a8a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x564c5f6b3fa5] 6: ceph-osd(+0x3f3106) [0x564c5f6b4106] 7: ceph-osd(+0x3e4d9c) [0x564c5f6a5d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x564c5fc176c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x564c5fc2171d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x564c5fc91061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x564c5fc9468b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x564c5fc95268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x564c5fc82fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x564c5fc83e33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x564c5fb8370d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x564c5f945592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x564c5f90d607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x564c5f903e65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x564c5f83fd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x564c5fa89a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x564c5f85a727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x564c5fd58d2a] 23: ceph-osd(+0xa982d4) [0x564c5fd592d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f251a89f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f251a83f450] 2024-04-01T23:17:52.332 INFO:scrape:1 jobs: ['7626050'] 2024-04-01T23:17:52.332 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton/{all/rebuild-mondb', 'supported-random-distro$/{centos_latest}}'] 2024-04-01T23:17:52.332 INFO:scrape: 2024-04-01T23:17:52.332 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55dc877670bc] 2024-04-01T23:17:52.332 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f00f2d5c520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55dc877670bc] 6: ceph-osd(+0x3f721d) [0x55dc8776721d] 7: ceph-osd(+0x3e9888) [0x55dc87759888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55dc87cb1c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55dc87cc605e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55dc87d272a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55dc87d3052b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55dc87d310a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55dc87d35611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55dc87d36433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55dc87c28c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55dc879de462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55dc879b496f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55dc879b84e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55dc878efaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55dc87b34799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55dc878fa793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55dc87df3313] 23: ceph-osd(+0xa83874) [0x55dc87df3874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f00f2daeb43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f00f2e40a00] 2024-04-01T23:17:52.332 INFO:scrape:1 jobs: ['7626167'] 2024-04-01T23:17:52.332 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-04-01T23:17:52.332 INFO:scrape: 2024-04-01T23:17:52.332 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ddb5e9ffa5] 2024-04-01T23:17:52.332 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f5176654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f51766a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ddb5e9ffa5] 6: ceph-osd(+0x3f3106) [0x55ddb5ea0106] 7: ceph-osd(+0x3e4d9c) [0x55ddb5e91d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ddb64036c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55ddb640d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55ddb647d061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ddb648068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ddb6481268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55ddb646efe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55ddb646fe33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55ddb636f70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55ddb6131592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55ddb60f9607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55ddb60efe65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ddb602bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ddb6275a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55ddb6046727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55ddb6544d2a] 23: ceph-osd(+0xa982d4) [0x55ddb65452d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f517669f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f517663f450] 2024-04-01T23:17:52.332 INFO:scrape:1 jobs: ['7626197'] 2024-04-01T23:17:52.333 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] 2024-04-01T23:17:52.333 INFO:scrape: 2024-04-01T23:17:52.333 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5652fc3240bc] 2024-04-01T23:17:52.333 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4bb8dde520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5652fc3240bc] 6: ceph-osd(+0x3f721d) [0x5652fc32421d] 7: ceph-osd(+0x3e9888) [0x5652fc316888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5652fc86ec30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5652fc88305e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5652fc8e42a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5652fc8ed52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5652fc8ee0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5652fc8f2611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5652fc8f3433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5652fc7e5c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5652fc59b462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5652fc57196f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5652fc5754e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5652fc4acaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5652fc6f1799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5652fc4b7793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5652fc9b0313] 23: ceph-osd(+0xa83874) [0x5652fc9b0874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4bb8e30b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4bb8ec2a00] 2024-04-01T23:17:52.333 INFO:scrape:1 jobs: ['7626186'] 2024-04-01T23:17:52.333 INFO:scrape:suites: ['mon_election/connectivity', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/wpq_default_shards', 'settings/optimized', 'ubuntu_latest', 'workloads/fio_4K_rand_read}'] 2024-04-01T23:17:52.333 INFO:scrape: 2024-04-01T23:17:52.333 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x558da2496fa5] 2024-04-01T23:17:52.333 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f13db454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f13db4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x558da2496fa5] 6: ceph-osd(+0x3f3106) [0x558da2497106] 7: ceph-osd(+0x3e4d9c) [0x558da2488d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x558da29fa6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x558da2a0471d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x558da2a74061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x558da2a7768b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x558da2a78268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x558da2a65fe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x558da2a66e33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x558da297f2a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x558da2964397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x558da279ea56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x558da26e8edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x558da2622d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x558da286ca19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x558da263d727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x558da2b3bd2a] 23: ceph-osd(+0xa982d4) [0x558da2b3c2d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f13db49f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f13db43f450] 2024-04-01T23:17:52.333 INFO:scrape:1 jobs: ['7626194'] 2024-04-01T23:17:52.333 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/small-objects}'] 2024-04-01T23:17:52.334 INFO:scrape: 2024-04-01T23:17:52.334 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a1ed8cf0bc] 2024-04-01T23:17:52.334 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1a0f925520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a1ed8cf0bc] 6: ceph-osd(+0x3f721d) [0x55a1ed8cf21d] 7: ceph-osd(+0x3e9888) [0x55a1ed8c1888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55a1ede19c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55a1ede2e05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55a1ede8f2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55a1ede9852b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55a1ede990a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55a1ede9d611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55a1ede9e433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55a1edd8ce3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55a1edd8e937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55a1edbab6f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55a1edb0374d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55a1eda57aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a1edc9c799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55a1eda62793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55a1edf5b313] 23: ceph-osd(+0xa83874) [0x55a1edf5b874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1a0f977b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f1a0fa09a00] 2024-04-01T23:17:52.334 INFO:scrape:1 jobs: ['7626156'] 2024-04-01T23:17:52.334 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_cls_all}'] 2024-04-01T23:17:52.334 INFO:scrape: 2024-04-01T23:17:52.334 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5557af04bfa5] 2024-04-01T23:17:52.334 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f7603054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f76030a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5557af04bfa5] 6: ceph-osd(+0x3f3106) [0x5557af04c106] 7: ceph-osd(+0x3e4d9c) [0x5557af03dd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5557af5af6c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5557af5b971d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5557af629061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5557af62c68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5557af62d268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5557af61afe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5557af61be33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5557af51b70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5557af2dd592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5557af2a5607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5557af29be65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5557af1d7d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5557af421a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5557af1f2727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5557af6f0d2a] 23: ceph-osd(+0xa982d4) [0x5557af6f12d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f760309f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f760303f450] 2024-04-01T23:17:52.334 INFO:scrape:1 jobs: ['7626057'] 2024-04-01T23:17:52.334 INFO:scrape:suites: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_latest}', 'thrashers/force-sync-many', 'workloads/snaps-few-objects}'] 2024-04-01T23:17:52.334 INFO:scrape: 2024-04-01T23:17:52.335 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c00b7330bc] 2024-04-01T23:17:52.335 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0ab3395520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c00b7330bc] 6: ceph-osd(+0x3f721d) [0x55c00b73321d] 7: ceph-osd(+0x3e9888) [0x55c00b725888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55c00bc7dc30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55c00bc9205e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55c00bcf32a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55c00bcfc52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55c00bcfd0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55c00bd01611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55c00bd02433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55c00bbf4c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55c00b9aa462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55c00b98096f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55c00b9844e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55c00b8bbaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c00bb00799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55c00b8c6793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55c00bdbf313] 23: ceph-osd(+0xa83874) [0x55c00bdbf874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0ab33e7b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f0ab3479a00] 2024-04-01T23:17:52.335 INFO:scrape:1 jobs: ['7626124'] 2024-04-01T23:17:52.335 INFO:scrape:suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}'] 2024-04-01T23:17:52.335 INFO:scrape: 2024-04-01T23:17:52.335 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55fa7f73b0bc] 2024-04-01T23:17:52.335 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2b45958520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55fa7f73b0bc] 6: ceph-osd(+0x3f721d) [0x55fa7f73b21d] 7: ceph-osd(+0x3e9888) [0x55fa7f72d888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55fa7fc85c30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55fa7fc9a05e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55fa7fcfb2a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55fa7fd0452b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55fa7fd050a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55fa7fd09611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55fa7fd0a433] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xadb) [0x55fa7fbf8e3b] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55fa7fbfa937] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55fa7fa176f6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55fa7f96f74d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55fa7f8c3aa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fa7fb08799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55fa7f8ce793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55fa7fdc7313] 23: ceph-osd(+0xa83874) [0x55fa7fdc7874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2b459aab43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2b45a3ca00] 2024-04-01T23:17:52.335 INFO:scrape:1 jobs: ['7626173'] 2024-04-01T23:17:52.335 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] 2024-04-01T23:17:52.335 INFO:scrape: 2024-04-01T23:17:52.335 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5634a157ffa5] 2024-04-01T23:17:52.336 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f484c454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f484c4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5634a157ffa5] 6: ceph-osd(+0x3f3106) [0x5634a1580106] 7: ceph-osd(+0x3e4d9c) [0x5634a1571d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5634a1ae36c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5634a1aed71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5634a1b5d061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5634a1b6068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5634a1b61268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5634a1b4efe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5634a1b4fe33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x5634a1a682a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x5634a1a4d397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x5634a1887a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x5634a17d1edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5634a170bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5634a1955a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5634a1726727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5634a1c24d2a] 23: ceph-osd(+0xa982d4) [0x5634a1c252d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f484c49f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f484c43f450] 2024-04-01T23:17:52.336 INFO:scrape:1 jobs: ['7626028'] 2024-04-01T23:17:52.336 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/redirect_promote_tests}'] 2024-04-01T23:17:52.336 INFO:scrape: 2024-04-01T23:17:52.336 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e40ecdffa5] 2024-04-01T23:17:52.336 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f208f254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f208f2a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e40ecdffa5] 6: ceph-osd(+0x3f3106) [0x55e40ece0106] 7: ceph-osd(+0x3e4d9c) [0x55e40ecd1d9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e40f2436c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55e40f24d71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55e40f2bd061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e40f2c068b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e40f2c1268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55e40f2aefe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55e40f2afe33] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x55e40f1c82a2] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55e40f1ad397] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55e40efe7a56] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x55e40ef31edd] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e40ee6bd27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e40f0b5a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55e40ee86727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55e40f384d2a] 23: ceph-osd(+0xa982d4) [0x55e40f3852d4] 24: /lib64/libc.so.6(+0x9f802) [0x7f208f29f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f208f23f450] 2024-04-01T23:17:52.336 INFO:scrape:1 jobs: ['7626063'] 2024-04-01T23:17:52.336 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/scrub_test}'] 2024-04-01T23:17:52.336 INFO:scrape: 2024-04-01T23:17:52.336 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5634436a20bc] 2024-04-01T23:17:52.336 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7abceee520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5634436a20bc] 6: ceph-osd(+0x3f721d) [0x5634436a221d] 7: ceph-osd(+0x3e9888) [0x563443694888] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x563443becc30] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x563443c0105e] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x563443c622a1] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x563443c6b52b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x563443c6c0a8] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x563443c70611] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x563443c71433] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x563443b63c9d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x563443919462] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5634438ef96f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5634438f34e9] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56344382aaa7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563443a6f799] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563443835793] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x563443d2e313] 23: ceph-osd(+0xa83874) [0x563443d2e874] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f7abcf40b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f7abcfd2a00] 2024-04-01T23:17:52.337 INFO:scrape:1 jobs: ['7626196'] 2024-04-01T23:17:52.337 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2024-04-01T23:17:52.337 INFO:scrape: 2024-04-01T23:17:52.337 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ff31f2dfa5] 2024-04-01T23:17:52.337 INFO:scrape:ceph version 19.0.0-1562-g41ed10b1 (41ed10b136a0de08323d8f65f00f42ebc2ccbcd8) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fb95e254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fb95e2a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ff31f2dfa5] 6: ceph-osd(+0x3f3106) [0x55ff31f2e106] 7: ceph-osd(+0x3e4d9c) [0x55ff31f1fd9c] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ff324916c0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55ff3249b71d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55ff3250b061] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ff3250e68b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ff3250f268] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55ff324fcfe5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55ff324fde33] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55ff323fd70d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55ff321bf592] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55ff32187607] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55ff3217de65] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ff320b9d27] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ff32303a19] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55ff320d4727] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55ff325d2d2a] 23: ceph-osd(+0xa982d4) [0x55ff325d32d4] 24: /lib64/libc.so.6(+0x9f802) [0x7fb95e29f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fb95e23f450] 2024-04-01T23:17:52.337 INFO:scrape:1 jobs: ['7626195'] 2024-04-01T23:17:52.337 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-small-objects-many-deletes}'] 2024-04-01T23:17:52.337 INFO:scrape: 2024-04-01T23:17:52.337 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-04-01T23:17:52.337 INFO:scrape:1 jobs: ['7626085'] 2024-04-01T23:17:52.338 INFO:scrape:suites: ['2-inject-leak/osd', 'centos_latest}', 'rados/valgrind-leaks/{1-start'] 2024-04-01T23:17:52.338 INFO:scrape: 2024-04-01T23:17:52.338 INFO:scrape:Dead: None 2024-04-01T23:17:52.338 INFO:scrape:1 jobs: ['7626065'] 2024-04-01T23:17:52.338 INFO:scrape: 2024-04-01T23:17:52.347 INFO:teuthology.results:Sending results to pdiazbou@redhat.com: Test Run: pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/ failed: 71 dead: 5 running: 0 waiting: 0 queued: 0 passed: 124 Fail ================================================================= [7626012] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:50:26 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626012/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626012/ "2024-04-01T11:00:00.000171+0000 mon.a (mon.0) 2524 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7626013] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 00:16:38 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626013/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626013/ sentry: https://sentry.ceph.com/organizations/ceph/?query=424cbe6a5d6849c994c9dfabdd29858a Command failed on smithi131 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626021] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:21:41 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626021/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626021/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f5c9b19d5084d7fb5cd5bc073a8f760 valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7626023] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:19:09 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626023/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626023/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ecce389644004c16ba35fd30b2f1c8aa Command failed on smithi073 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626025] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:16:43 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626025/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626025/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e973c661ff9d4d84bd5d2f9d1e1f7ff1 Command failed on smithi080 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626028] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:09:36 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626028/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626028/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5fbf1f7da9204f67b4917797f7e4838b Command failed on smithi088 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' [7626029] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/repair_test} ----------------------------------------------------------------- time: 00:07:51 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626029/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626029/ sentry: https://sentry.ceph.com/organizations/ceph/?query=14a138da43c046e4a929a294ab776dce Command failed on smithi175 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' [7626032] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:15:06 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626032/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626032/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d3b4fd12aed4c8c9482a825df22173b Command failed on smithi042 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626033] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:17:05 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626033/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626033/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2b4595648a044d68dde0fb4fa89f3e1 Command failed on smithi121 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626042] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:31 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626042/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626042/ "2024-04-01T10:27:51.075991+0000 mon.smithi082 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7626044] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:36:14 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626044/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626044/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6dfcda2f62854c6ba5e99aeecec82790 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*/1:-*SyntheticMatrixC* --gtest_catch_exceptions=0\'' [7626046] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} ----------------------------------------------------------------- time: 00:14:27 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626046/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626046/ sentry: https://sentry.ceph.com/organizations/ceph/?query=36859e425efe4c75ad335b727f5c5868 timeout expired in wait_for_all_osds_up [7626050] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:31:43 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626050/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626050/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8c055548c83419fb5517b9b09f2fc12 reached maximum tries (201) after waiting for 1200 seconds [7626051] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:14:44 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626051/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626051/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe5eb8928fb845cabb503ec14ce7a92e Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626052] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:07:43 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626052/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626052/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5705003433d24d3485c645a2a0a278f3 Command failed on smithi181 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' [7626054] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:17:16 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626054/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626054/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f4407ffb59b40269f2209f3c1ffe2d3 Command failed on smithi038 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626057] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:07:17 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626057/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626057/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b779798bf02348648312c5a3d3d9f412 Command failed on smithi084 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' [7626063] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/scrub_test} ----------------------------------------------------------------- time: 00:19:11 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626063/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626063/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1bb664c4b43c47d5859dc7d1b4b610c3 Command failed on smithi045 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626066] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:14:32 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626066/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626066/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee748c5ec13240048e37024d6e0c5604 Command failed on smithi105 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626069] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:14:05 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626069/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626069/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5dfc243fcf96470ebafef982ae4aaf42 Command failed on smithi175 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626070] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:16:33 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626070/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626070/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0c5dbdd9fd344fd9bfdeae8729167e1 Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626072] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:00 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626072/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626072/ "2024-04-01T10:39:25.959540+0000 mon.smithi080 (mon.0) 258 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7626074] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:14:45 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626074/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626074/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b06e565c9b8843e5a2bd776fa04ea40f Command failed on smithi135 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626075] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:07:40 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626075/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626075/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0b79e0d2e6264e669c28403c0828f464 Command failed on smithi049 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' [7626077] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:08:12 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626077/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626077/ sentry: https://sentry.ceph.com/organizations/ceph/?query=26e3b70348874d0ebd5b16a2cb89de5e Command failed on smithi186 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8' [7626080] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:15:37 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626080/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626080/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f700918c41047358d4c8d5e9d6236c3 Command failed on smithi117 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626082] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/insights} ----------------------------------------------------------------- time: 00:40:15 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626082/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626082/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c4b34f0c9b24433aa99f7564c1aad3c wait_for_clean: failed before timeout expired [7626085] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} ----------------------------------------------------------------- time: 00:22:14 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626085/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626085/ reached maximum tries (51) after waiting for 300 seconds [7626093] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:12:39 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626093/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626093/ "2024-04-01T10:55:29.605777+0000 mon.smithi121 (mon.0) 258 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7626095] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:16:50 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626095/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626095/ sentry: https://sentry.ceph.com/organizations/ceph/?query=683c1c27f4d24b63a200e1c80a977b00 Command failed on smithi012 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626097] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:16:50 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626097/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626097/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d829d1caef7941efafe7b2742023295a Command failed on smithi162 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626100] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:10:10 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626100/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626100/ sentry: https://sentry.ceph.com/organizations/ceph/?query=151f2e502a9b42be9aac34948e08afc4 Command failed on smithi046 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' [7626105] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/pool-create-delete} ----------------------------------------------------------------- time: 00:18:53 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626105/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f9c3aed929e47ce97e374168a101ff6 Command failed on smithi003 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626106] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:14:35 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626106/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626106/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb02db8d28454455ac9ad73f80fd7ba7 Command failed on smithi106 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626112] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:51:12 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626112/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626112/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7626121] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:19:15 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626121/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626121/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd6be3e876454372baa5271170e58f05 Command failed on smithi154 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626124] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:17:31 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626124/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626124/ sentry: https://sentry.ceph.com/organizations/ceph/?query=571672f07015460c8f1d1b6af32873e8 Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626125] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:20:01 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626125/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626125/ "2024-04-01T11:14:54.456109+0000 mon.smithi080 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7626126] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:10:05 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626126/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626126/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13d974db0c8d40cc84006826f63912af Command failed on smithi069 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' [7626129] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:13:59 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626129/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626129/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5981f6a22b75472aa989bff99e3be75c Command failed on smithi084 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626130] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:23:14 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626130/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626130/ sentry: https://sentry.ceph.com/organizations/ceph/?query=870746f89d2146989e3aaaefceeb5878 Command failed on smithi086 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626135] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:07:31 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626135/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626135/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f5c234a642484cb5b61ddbd623bf365c Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7626140] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:10:58 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626140/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626140/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aa6ed4622aa24a82b3c201065368f6cc Command failed on smithi070 with status 5: 'sudo systemctl stop ceph-966734f0-f01a-11ee-b647-cb9ed24678a4@mon.a' [7626142] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:17:27 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626142/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626142/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e016c3264152452185834f6b837b6ed1 Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626144] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:14:32 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626144/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626144/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1fd71dc052c14be0bc10b7e27bd0082d Command failed on smithi117 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json' [7626145] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:09:19 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626145/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626145/ sentry: https://sentry.ceph.com/organizations/ceph/?query=10d2ef83f578455aa1f1a526646c6b7e Command failed (workunit test cephadm/test_cephadm.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41ed10b136a0de08323d8f65f00f42ebc2ccbcd8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7626148] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:07:52 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626148/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626148/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8777de13aea04fd8b5859dd23ef41179 Command failed on smithi052 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' [7626150] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:17:56 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626150/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626150/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3723380a39054716875aa265e4c7b974 Command failed on smithi046 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626151] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:19:12 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626151/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626151/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a24caf5c6244f6fb9d888e62b225ac5 Command failed on smithi113 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626152] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:40 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626152/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626152/ "2024-04-01T11:30:09.086033+0000 mon.smithi007 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7626156] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} ----------------------------------------------------------------- time: 00:17:18 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626156/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626156/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ff8b0bbcf9f4444a3c834b51bb630e8 Command failed on smithi175 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626159] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:07:04 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626159/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626159/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2bb0377bec574dfb82e01659e6e1478d Command failed on smithi098 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' [7626160] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_latest} tasks/per_module_finisher_stats} ----------------------------------------------------------------- time: 00:38:00 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626160/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626160/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39121ed751704cdab93d11a4999b0db4 wait_for_clean: failed before timeout expired [7626167] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:19:24 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626167/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626167/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c1972b93b0144858dd9c5243f70ce81 Command failed on smithi123 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626169] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:18:44 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626169/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626169/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f025b068ccc94af7bb5427a8ebbfc375 Command failed on smithi165 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626173] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:17:20 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626173/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626173/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34dad9c80a384fc0ab080c36cf32863f Command failed on smithi047 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626175] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:08:20 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626175/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626175/ sentry: https://sentry.ceph.com/organizations/ceph/?query=143d5e2f7a204510b4d3c53f5a892615 Command failed on smithi107 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' [7626176] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:17:08 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626176/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29c8d686ee234fab866256fedbf0f22c Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626177] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:07 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626177/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626177/ "2024-04-01T11:42:40.970404+0000 mon.smithi094 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7626180] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} ----------------------------------------------------------------- time: 01:33:38 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626180/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626180/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3359325aeb34e4982db56531f3bb995 Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41ed10b136a0de08323d8f65f00f42ebc2ccbcd8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- repair.sh' [7626184] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:19:49 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626184/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626184/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8b3ad67348442debeb251ba849162b6 Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi006 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41ed10b136a0de08323d8f65f00f42ebc2ccbcd8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' [7626186] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} ----------------------------------------------------------------- time: 00:21:32 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626186/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4e3edd1266e462fba33168068635602 Command failed on smithi067 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626189] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} ----------------------------------------------------------------- time: 00:10:08 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626189/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cec3b97bc4a840cd91d9359fc02f2b9a Command failed on smithi040 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' [7626191] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:10:12 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626191/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626191/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08ddd88e4de941ac965db7d21f9a87c3 Command failed on smithi043 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' [7626194] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:14:59 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626194/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626194/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2fee8edea35e4efe83a857d106d235ff Command failed on smithi029 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626195] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:14:42 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626195/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f10a0fb2e5c4c198faba4b276edbf99 Command failed on smithi102 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json' [7626196] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:18:10 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626196/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626196/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03f7038bc832433b9d1a8fbaca5a498b Command failed on smithi050 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626197] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:07:31 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626197/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626197/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba0cc7a18a10458fb22d0927e04b81e8 Command failed on smithi026 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' [7626199] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:20:33 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626199/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626199/ "2024-04-01T12:01:54.621424+0000 mon.smithi137 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7626201] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:17:14 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626201/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626201/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0b8f3a2ac8ec49c7a0744bafb15fe50b Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7626204] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:14:27 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626204/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e540771482e43a0abb0efec646dfb8b No module named 'tasks.ceph' Dead ================================================================= [7626010] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626010/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626010/ hit max job timeout [7626048] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626048/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626048/ hit max job timeout [7626065] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626065/ Error reimaging machines: Failed to power on smithi143 [7626119] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626119/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626119/ hit max job timeout [7626137] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626137/ log: http://qa-proxy.ceph.com/teuthology/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626137/ hit max job timeout Pass ================================================================= [7626011] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} time: 00:27:04 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626011/ [7626014] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:31:17 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626014/ [7626015] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:23:55 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626015/ [7626016] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:54 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626016/ [7626017] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:21 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626017/ [7626018] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:10:27 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626018/ [7626019] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/rados_api_tests} time: 00:32:52 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626019/ [7626020] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_host_drain} time: 00:15:23 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626020/ [7626022] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:35:42 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626022/ [7626024] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:27 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626024/ [7626026] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:11:54 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626026/ [7626027] rados/singleton/{all/radostool mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:13:29 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626027/ [7626030] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:23:06 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626030/ [7626031] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:52 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626031/ [7626034] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:39 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626034/ [7626035] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:22:39 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626035/ [7626036] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:14:55 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626036/ [7626037] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:28 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626037/ [7626038] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:19:03 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626038/ [7626039] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:27:34 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626039/ [7626040] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:19:20 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626040/ [7626041] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:24:45 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626041/ [7626043] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/failover} time: 00:12:58 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626043/ [7626045] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:22:12 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626045/ [7626047] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:17 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626047/ [7626049] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:24:58 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626049/ [7626053] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon} time: 01:02:42 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626053/ [7626055] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_orch_cli} time: 00:14:18 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626055/ [7626056] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:38 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626056/ [7626058] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:14:47 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626058/ [7626059] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-radosbench} time: 00:37:27 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626059/ [7626060] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:55 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626060/ [7626061] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:13:07 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626061/ [7626062] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} time: 00:22:33 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626062/ [7626064] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:18 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626064/ [7626067] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:14 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626067/ [7626068] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:24:50 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626068/ [7626071] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:11:20 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626071/ [7626073] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:12:18 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626073/ [7626076] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:47 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626076/ [7626078] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:38 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626078/ [7626079] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_orch_cli_mon} time: 00:24:37 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626079/ [7626081] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:08 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626081/ [7626083] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:34:51 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626083/ [7626084] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:10:33 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626084/ [7626086] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/dedup-io-mixed} time: 00:11:01 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626086/ [7626087] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/dedup-io-snaps} time: 00:12:56 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626087/ [7626088] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:21:18 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626088/ [7626089] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} time: 00:28:36 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626089/ [7626090] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:07:46 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626090/ [7626091] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rados_api_tests} time: 00:26:44 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626091/ [7626092] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:23 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626092/ [7626094] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:12:33 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626094/ [7626096] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill} time: 02:25:59 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626096/ [7626098] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:23:43 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626098/ [7626099] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:35 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626099/ [7626101] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:15:13 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626101/ [7626102] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:42 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626102/ [7626103] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:08:50 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626103/ [7626104] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 02:08:26 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626104/ [7626107] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-balanced} time: 00:19:59 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626107/ [7626108] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:12 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626108/ [7626109] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:16:22 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626109/ [7626110] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:24:40 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626110/ [7626111] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/set-chunks-read} time: 00:20:15 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626111/ [7626113] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:28 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626113/ [7626114] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} time: 00:26:24 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626114/ [7626115] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:26:21 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626115/ [7626116] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} time: 00:22:40 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626116/ [7626117] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_adoption} time: 00:12:12 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626117/ [7626118] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/e2e} time: 00:25:00 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626118/ [7626120] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{centos_latest} tasks/module_selftest} time: 00:23:40 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626120/ [7626122] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:11 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626122/ [7626123] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:16:27 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626123/ [7626127] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:22:23 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626127/ [7626128] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:53 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626128/ [7626131] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:22 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626131/ [7626132] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:11 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626132/ [7626133] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:32:54 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626133/ [7626134] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:12:28 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626134/ [7626136] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:20:11 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626136/ [7626138] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-big} time: 00:16:13 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626138/ [7626139] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} time: 02:30:23 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626139/ [7626141] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-small} time: 00:14:29 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626141/ [7626143] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:59 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626143/ [7626146] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:08:03 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626146/ [7626147] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925} time: 00:11:46 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626147/ [7626149] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read} time: 00:19:48 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626149/ [7626153] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache} time: 00:15:06 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626153/ [7626154] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:15:22 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626154/ [7626155] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:11:08 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626155/ [7626157] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:23:05 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626157/ [7626158] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/dedup-io-mixed} time: 00:10:24 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626158/ [7626161] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:29 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626161/ [7626162] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-snaps} time: 00:14:20 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626162/ [7626163] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos} time: 00:06:42 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626163/ [7626164] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{ubuntu_latest}} time: 00:09:38 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626164/ [7626165] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:30:56 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626165/ [7626166] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:18:14 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626166/ [7626168] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:04 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626168/ [7626170] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:06 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626170/ [7626171] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:24:19 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626171/ [7626172] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:34:58 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626172/ [7626174] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:17 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626174/ [7626178] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:18:09 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626178/ [7626179] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:15:11 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626179/ [7626181] rados/singleton/{all/deduptool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:13:18 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626181/ [7626182] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:24:59 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626182/ [7626183] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:11:38 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626183/ [7626185] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:14:44 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626185/ [7626187] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:51 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626187/ [7626188] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:22:03 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626188/ [7626190] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:11:24 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626190/ [7626192] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_extra_daemon_features} time: 00:13:24 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626192/ [7626193] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/rados_api_tests} time: 00:28:29 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626193/ [7626198] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/progress} time: 00:20:25 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626198/ [7626200] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:22:30 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626200/ [7626202] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:40 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626202/ [7626203] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:26:50 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626203/ [7626205] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:17:45 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626205/ [7626206] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain} time: 00:16:38 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626206/ [7626207] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:32:15 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626207/ [7626208] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:06 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626208/ [7626209] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:17:41 info: https://pulpito.ceph.com/pdiazbou-2024-03-27_11:08:38-rados-wip-pereman2-blob_use_tracker_t-power-of-two-distro-default-smithi/7626209/