2024-03-19T04:57:14.658 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-19T04:57:14.658 INFO:teuthology.results:subset: '433/512' 2024-03-19T04:57:14.658 INFO:teuthology.results:seed: '3005' 2024-03-19T07:48:08.790 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-19T07:48:08.815 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-19T19:04:05.143 INFO:teuthology.results:Tests finished! gathering results... 2024-03-19T19:04:12.817 INFO:scrape:Found 84 jobs 2024-03-19T19:04:14.110 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.110 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.110 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.111 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.112 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.112 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.113 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.113 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.238 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.238 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.238 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.238 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.240 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.240 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.240 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.240 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.363 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.364 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.364 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.364 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.365 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.366 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.366 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:14.366 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.094 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.094 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.094 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.095 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.245 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.245 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.245 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.245 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.323 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.323 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.323 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.323 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.402 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.403 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.403 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.403 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.480 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.481 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.481 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.481 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.556 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.557 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.557 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.557 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.636 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.636 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.636 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:34.637 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:47.600 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610554/teuthology.log 2024-03-19T19:04:56.572 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:56.573 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:56.573 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:56.573 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:56.882 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:56.883 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:56.883 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:56.883 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.126 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.127 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.127 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.127 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.374 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.374 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.375 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.375 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.619 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.619 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.620 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.620 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.858 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.859 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.859 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:57.859 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.106 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.106 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.106 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.106 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.604 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.605 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.605 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.606 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.606 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.606 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.606 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.606 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.723 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.724 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.724 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.724 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.817 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.817 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.817 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.817 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.983 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.983 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.983 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.984 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.984 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.984 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.984 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:58.984 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.053 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.053 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.053 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.053 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.055 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.056 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.056 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.056 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.221 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.221 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.221 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.221 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.221 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.221 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.222 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.222 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.286 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.286 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.287 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.287 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.289 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.289 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.289 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.289 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.468 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.468 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.469 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.469 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.469 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.469 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.469 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.469 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.541 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.541 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.541 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.541 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.544 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.544 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.544 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.544 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.717 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.848 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.848 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.849 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.849 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.849 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.849 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.849 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.931 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.932 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.932 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.932 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.934 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.934 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.934 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:04:59.935 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.120 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.121 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.121 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.121 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.121 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.121 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.121 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.121 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.190 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.190 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.190 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.190 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.192 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.193 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.193 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.193 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.350 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.350 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.351 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.351 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.351 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.351 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.351 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.351 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.413 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.413 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.413 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.413 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.415 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.415 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.415 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:00.415 WARNING:scrape:Saw end of BT but not start 2024-03-19T19:05:08.685 INFO:scrape:Found 21 distinct failure reasons 2024-03-19T19:05:08.685 INFO:scrape:Dead: b'2024-03-19T18:58:12.305 DEBUG:teuthology.exit:Finished running handlers' 2024-03-19T19:05:08.685 INFO:scrape:1 jobs: ['7610575'] 2024-03-19T19:05:08.685 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2024-03-19T19:05:08.686 INFO:scrape: 2024-03-19T19:05:08.686 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-19T19:05:08.686 INFO:scrape:3 jobs: ['7610614', '7610611', '7610538'] 2024-03-19T19:05:08.686 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'replication/default}', 'session_timeout}', 'standby-replay'] 2024-03-19T19:05:08.686 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-03-19T19:05:08.686 INFO:scrape: 2024-03-19T19:05:08.686 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-19T19:05:08.686 INFO:scrape:5 jobs: ['7610589', '7610534', '7610557', '7610605', '7610573'] 2024-03-19T19:05:08.687 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-03-19T19:05:08.687 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-03-19T19:05:08.687 INFO:scrape: 2024-03-19T19:05:08.687 INFO:scrape:Failure: Command failed on smithi148 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.36 deep-scrub' 2024-03-19T19:05:08.687 INFO:scrape:4 jobs: ['7610583', '7610563', '7610539', '7610603'] 2024-03-19T19:05:08.687 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2024-03-19T19:05:08.687 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2024-03-19T19:05:08.687 INFO:scrape: 2024-03-19T19:05:08.688 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi044 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-19T19:05:08.688 INFO:scrape:4 jobs: ['7610549', '7610584', '7610566', '7610602'] 2024-03-19T19:05:08.688 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-03-19T19:05:08.689 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{balancer/automatic', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-03-19T19:05:08.689 INFO:scrape: 2024-03-19T19:05:08.689 INFO:scrape:Failure: "2024-03-19T07:34:00.182604+0000 mds.c (mds.2) 23 : cluster [WRN] Scrub error on inode 0x30000002965 (/volumes/qa/sv_1/f41e6e49-d9b5-4b54-b194-08df95a6c68e/client.0/tmp/t/linux-6.5.11/arch) see mds.c log and `damage ls` output for details" in cluster log 2024-03-19T19:05:08.689 INFO:scrape:3 jobs: ['7610576', '7610540', '7610555'] 2024-03-19T19:05:08.689 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2024-03-19T19:05:08.689 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-03-19T19:05:08.689 INFO:scrape: 2024-03-19T19:05:08.689 INFO:scrape:Failure: "2024-03-19T07:00:00.000215+0000 mon.a (mon.0) 679 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg inactive" in cluster log 2024-03-19T19:05:08.690 INFO:scrape:5 jobs: ['7610560', '7610574', '7610545', '7610592', '7610542'] 2024-03-19T19:05:08.690 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}'] 2024-03-19T19:05:08.690 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'export-check', 'frag', 'fs/functional/{begin/{0-install', 'fs/nfs/{cluster/{1-node}', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/ignorelist_health', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'subvol_versions/create_subvol_version_v1', 'supported-random-distros$/{centos_latest}', 'tasks/forward-scrub}', 'tasks/nfs}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-03-19T19:05:08.690 INFO:scrape: 2024-03-19T19:05:08.690 INFO:scrape:Failure: "2024-03-19T05:29:23.760001+0000 mds.b (mds.1) 1 : cluster [WRN] evicting unresponsive client smithi081:x (15471), after 300.08 seconds" in cluster log 2024-03-19T19:05:08.690 INFO:scrape:7 jobs: ['7610533', '7610564', '7610567', '7610543', '7610558', '7610556', '7610537'] 2024-03-19T19:05:08.690 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/random', 'session_timeout}', 'standby-replay'] 2024-03-19T19:05:08.691 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iozone}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2024-03-19T19:05:08.691 INFO:scrape: 2024-03-19T19:05:08.691 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-03-19T19:05:08.691 INFO:scrape:1 jobs: ['7610580'] 2024-03-19T19:05:08.691 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2024-03-19T19:05:08.691 INFO:scrape: 2024-03-19T19:05:08.692 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-19T19:05:08.692 INFO:scrape:2 jobs: ['7610597', '7610612'] 2024-03-19T19:05:08.692 INFO:scrape:suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-19T19:05:08.692 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-19T19:05:08.692 INFO:scrape: 2024-03-19T19:05:08.692 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-03-19T19:05:08.692 INFO:scrape:1 jobs: ['7610546'] 2024-03-19T19:05:08.693 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-19T19:05:08.693 INFO:scrape: 2024-03-19T19:05:08.693 INFO:scrape:Failure: "2024-03-19T08:21:40.691957+0000 mds.b (mds.0) 189 : cluster [WRN] Scrub error on inode 0x10000003804 (/volumes/qa/sv_0/32656221-f637-44e3-81ee-40fb5a32bc92/client.0/tmp/clients/client4/~dmtmp/WORD) see mds.b log and `damage ls` output for details" in cluster log 2024-03-19T19:05:08.693 INFO:scrape:1 jobs: ['7610599'] 2024-03-19T19:05:08.693 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}'] 2024-03-19T19:05:08.694 INFO:scrape: 2024-03-19T19:05:08.694 INFO:scrape:Failure: timeout expired in wait_until_healthy 2024-03-19T19:05:08.694 INFO:scrape:1 jobs: ['7610552'] 2024-03-19T19:05:08.694 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}'] 2024-03-19T19:05:08.694 INFO:scrape: 2024-03-19T19:05:08.694 INFO:scrape:Crash: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-03-19T19:05:08.694 INFO:scrape:ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x5791520] 2: (QuiesceDbManager::leader_bootstrap(std::queue > >&&, std::chrono::duration >&)+0x1d3) [0x569c53] 3: (QuiesceDbManager::quiesce_db_thread_main()+0xcdd) [0x566e5d] 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x57e3b43] 2024-03-19T19:05:08.694 INFO:scrape:1 jobs: ['7610594'] 2024-03-19T19:05:08.694 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-03-19T19:05:08.695 INFO:scrape: 2024-03-19T19:05:08.695 INFO:scrape:Failure: Command failed on smithi007 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bce2bae8-e5be-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph orch ps'" 2024-03-19T19:05:08.695 INFO:scrape:1 jobs: ['7610565'] 2024-03-19T19:05:08.695 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-19T19:05:08.695 INFO:scrape: 2024-03-19T19:05:08.695 INFO:scrape:Dead: None 2024-03-19T19:05:08.695 INFO:scrape:1 jobs: ['7610554'] 2024-03-19T19:05:08.695 INFO:scrape: 2024-03-19T19:05:08.695 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-19T19:05:08.695 INFO:scrape:1 jobs: ['7610587'] 2024-03-19T19:05:08.696 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-19T19:05:08.696 INFO:scrape: 2024-03-19T19:05:08.696 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi189 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' 2024-03-19T19:05:08.696 INFO:scrape:2 jobs: ['7610544', '7610541'] 2024-03-19T19:05:08.696 INFO:scrape:suites intersection: [] 2024-03-19T19:05:08.696 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-mount', '2-logrotate}', '2-scrub/no', '2-workload/dbench', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/single-container-host}', 'export-check', 'frag', 'fs/cephadm/multivolume/{0-start', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2024-03-19T19:05:08.696 INFO:scrape: 2024-03-19T19:05:08.696 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-03-19T19:05:08.697 INFO:scrape:1 jobs: ['7610609'] 2024-03-19T19:05:08.697 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-03-19T19:05:08.697 INFO:scrape: 2024-03-19T19:05:08.697 INFO:scrape:Failure: valgrind error: Leak_StillReachable malloc malloc strdup 2024-03-19T19:05:08.697 INFO:scrape:1 jobs: ['7610559'] 2024-03-19T19:05:08.697 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2024-03-19T19:05:08.698 INFO:scrape: 2024-03-19T19:05:08.698 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) 2024-03-19T19:05:08.698 INFO:scrape:1 jobs: ['7610547'] 2024-03-19T19:05:08.698 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2024-03-19T19:05:08.698 INFO:scrape: 2024-03-19T19:05:08.704 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/ failed: 45 dead: 2 running: 0 waiting: 0 queued: 0 passed: 37 Fail ================================================================= [7610533] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:31:34 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610533/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610533/ "2024-03-19T05:29:23.760001+0000 mds.b (mds.1) 1 : cluster [WRN] evicting unresponsive client smithi081:x (15471), after 300.08 seconds" in cluster log [7610534] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:37:00 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610534/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610534/ reached maximum tries (51) after waiting for 300 seconds [7610537] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:16:12 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610537/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610537/ "2024-03-19T06:33:41.425030+0000 mds.b (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi019:x (15360), after 301.73 seconds" in cluster log [7610538] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:56:36 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610538/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610538/ error during scrub thrashing: rank damage found: {'backtrace'} [7610539] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:22:30 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610539/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610539/ Command failed on smithi081 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.30 deep-scrub' [7610540] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:38:20 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610540/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610540/ "2024-03-19T06:49:01.860990+0000 mds.b (mds.0) 62 : cluster [WRN] Scrub error on inode 0x2000000337d (/volumes/qa/sv_1/3f009418-9f35-4703-9a0f-7dc0 f8f6746d/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-84) see mds.b log and `damage ls` output for details" in cluster log [7610541] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:35:10 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610541/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610541/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94d36ba1f30a4790a665843b737c6784 Command failed (workunit test suites/dbench.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7610542] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:00:20 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610542/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610542/ "2024-03-19T06:40:00.000097+0000 mon.a (mon.0) 1095 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem is online with fewer MDS than max_mds" in cluster log [7610543] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:32:58 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610543/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610543/ "2024-03-19T06:48:32.719019+0000 mds.b (mds.2) 1 : cluster [WRN] evicting unresponsive client smithi001:x (15297), after 300.889 seconds" in cluster log [7610544] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:35:10 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610544/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610544/ sentry: https://sentry.ceph.com/organizations/ceph/?query=385d28d42ec34311bb2b14942b3533d7 Command failed (workunit test suites/dbench.sh) on smithi189 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7610545] fs/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:34:45 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610545/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610545/ "2024-03-19T06:38:11.133196+0000 mon.a (mon.0) 490 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7610546] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:35:48 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610546/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610546/ sentry: https://sentry.ceph.com/organizations/ceph/?query=045ea143b27f470784fc153ea0868678 Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7610547] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:27:51 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610547/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610547/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8bb90975846c492480c290ff3fb7e89f Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) [7610549] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:31:35 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610549/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610549/ sentry: https://sentry.ceph.com/organizations/ceph/?query=622408a9c4084228b6ac49b29157366e Command failed (workunit test suites/fsx.sh) on smithi044 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7610552] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:20:38 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610552/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610552/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ab4b13a0cdd48ccb867992ba4953d9e timeout expired in wait_until_healthy [7610555] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:03:49 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610555/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610555/ "2024-03-19T07:17:36.978983+0000 mds.b (mds.0) 94 : cluster [WRN] Scrub error on inode 0x10000003331 (/volumes/qa/sv_1/c5bc364a-1908-4e0c-b024-638f 98ec6eab/client.0/tmp/payload.1/multiple_rsync_payload.193727/modules/5.14. 0-239.el9.x86_64/kernel/drivers/platform/x86/intel) see mds.b log and `damage ls` output for details" in cluster log [7610556] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:57:38 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610556/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610556/ "2024-03-19T07:06:51.545027+0000 mds.b (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi042:x (15288), after 303.306 seconds" in cluster log [7610557] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:38:06 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610557/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610557/ reached maximum tries (51) after waiting for 300 seconds [7610558] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:37:25 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610558/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610558/ "2024-03-19T07:06:19.924323+0000 mds.b (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi006:x (15375), after 301.525 seconds" in cluster log [7610559] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:44:55 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610559/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610559/ valgrind error: Leak_StillReachable malloc malloc strdup [7610560] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:56:19 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610560/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610560/ "2024-03-19T07:00:00.000215+0000 mon.a (mon.0) 679 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg inactive" in cluster log [7610563] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:20:56 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610563/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610563/ Command failed on smithi067 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.27 deep-scrub' [7610564] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:32:25 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610564/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610564/ "2024-03-19T07:19:09.153970+0000 mds.f (mds.4) 1 : cluster [WRN] evicting unresponsive client smithi066:x (15333), after 300.476 seconds" in cluster log [7610565] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:53:25 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610565/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610565/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15460cc11c694cd990e60489a60a1032 Command failed on smithi007 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bce2bae8-e5be-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph orch ps'" [7610566] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:30:00 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610566/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610566/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f7210c7765f422ea44cb5a0ca132829 Command failed (workunit test suites/fsx.sh) on smithi046 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7610567] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:33:53 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610567/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610567/ "2024-03-19T07:18:48.538829+0000 mds.h (mds.2) 1 : cluster [WRN] evicting unresponsive client smithi081:x (15282), after 301.223 seconds" in cluster log [7610573] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:39:48 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610573/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610573/ reached maximum tries (51) after waiting for 300 seconds [7610574] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} ----------------------------------------------------------------- time: 00:18:11 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610574/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610574/ "2024-03-19T07:12:21.916356+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log [7610576] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:33:50 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610576/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610576/ "2024-03-19T07:34:00.182604+0000 mds.c (mds.2) 23 : cluster [WRN] Scrub error on inode 0x30000002965 (/volumes/qa/sv_1/f41e6e49-d9b5-4b54-b194-08df 95a6c68e/client.0/tmp/t/linux-6.5.11/arch) see mds.c log and `damage ls` output for details" in cluster log [7610580] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:38:07 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610580/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610580/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0b1b68b1f6e4e37a89f9e863739eee0 Command failed (workunit test suites/ffsb.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7610583] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:08 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610583/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610583/ Command failed on smithi148 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.36 deep-scrub' [7610584] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:23:38 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610584/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610584/ sentry: https://sentry.ceph.com/organizations/ceph/?query=11c346cce5a846e395797fa4b554f10e Command failed (workunit test suites/fsx.sh) on smithi003 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7610587] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:12:47 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610587/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610587/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9639ade369e473080154ebb2807cb48 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7610589] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:52:31 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610589/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610589/ reached maximum tries (51) after waiting for 300 seconds [7610592] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:36:24 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610592/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610592/ "2024-03-19T07:40:00.000196+0000 mon.a (mon.0) 439 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log [7610594] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:39:31 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610594/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610594/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b90d3aa5f73c4131ab804388eec7ef39 valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun [7610597] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:25 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610597/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610597/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33d599676b4d4587baf802f7a6e08497 Command failed (workunit test suites/fsstress.sh) on smithi063 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7610599] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:03:33 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610599/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610599/ "2024-03-19T08:21:40.691957+0000 mds.b (mds.0) 189 : cluster [WRN] Scrub error on inode 0x10000003804 (/volumes/qa/sv_0/32656221-f637-44e3-81ee-40fb 5a32bc92/client.0/tmp/clients/client4/~dmtmp/WORD) see mds.b log and `damage ls` output for details" in cluster log [7610602] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:33:04 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610602/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610602/ sentry: https://sentry.ceph.com/organizations/ceph/?query=81412ebb88ad4e478610980a7f906eee Command failed (workunit test suites/fsx.sh) on smithi026 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7610603] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:08 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610603/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610603/ Command failed on smithi038 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.2c deep-scrub' [7610605] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:53:31 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610605/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610605/ reached maximum tries (51) after waiting for 300 seconds [7610609] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:13:24 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610609/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610609/ valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun [7610611] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:04:51 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610611/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610611/ error during scrub thrashing: rank damage found: {'backtrace'} [7610612] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:35:59 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610612/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610612/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a7dcfb91f61748429f166c19e706fe0f Command failed (workunit test suites/fsstress.sh) on smithi191 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a76be9e33cc5b22f5135f29be5c73cd2ad978d87 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7610614] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:39:32 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610614/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610614/ error during scrub thrashing: rank damage found: {'backtrace'} Dead ================================================================= [7610554] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610554/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610554/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7610575] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610575/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610575/ hit max job timeout Pass ================================================================= [7610532] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:48:27 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610532/ [7610535] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:23:39 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610535/ [7610536] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:28:02 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610536/ [7610548] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:29:18 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610548/ [7610550] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:25:00 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610550/ [7610551] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:33:05 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610551/ [7610553] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:31:41 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610553/ [7610561] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:50:31 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610561/ [7610562] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} time: 01:14:08 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610562/ [7610568] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:33:02 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610568/ [7610569] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:35:04 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610569/ [7610570] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:29:31 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610570/ [7610571] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} time: 00:31:53 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610571/ [7610572] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:32:45 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610572/ [7610577] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 00:40:41 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610577/ [7610578] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:35:57 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610578/ [7610579] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:12:36 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610579/ [7610581] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:24:56 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610581/ [7610582] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:32:58 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610582/ [7610585] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:36:35 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610585/ [7610586] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:32:35 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610586/ [7610588] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:47:41 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610588/ [7610590] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:51:19 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610590/ [7610591] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:23:26 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610591/ [7610593] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:23:20 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610593/ [7610595] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} time: 00:51:06 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610595/ [7610596] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 01:02:23 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610596/ [7610598] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:44:44 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610598/ [7610600] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:36:42 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610600/ [7610601] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:24:49 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610601/ [7610604] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:37:08 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610604/ [7610606] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:51:06 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610606/ [7610607] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:32:37 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610607/ [7610608] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:23:17 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610608/ [7610610] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:27:39 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610610/ [7610613] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:55:43 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610613/ [7610615] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:39:16 info: https://pulpito.ceph.com/pdonnell-2024-03-19_04:56:42-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7610615/