2024-03-01T17:29:16.897 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-01T17:29:16.898 INFO:teuthology.results:subset: '468/512'
2024-03-01T17:29:16.898 INFO:teuthology.results:seed: '7935'
2024-03-01T21:39:15.122 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-01T21:39:15.167 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-02T09:24:23.311 INFO:teuthology.results:Tests finished! gathering results...
2024-03-02T09:24:46.027 INFO:scrape:Found 245 jobs
2024-03-02T09:25:04.406 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:04.407 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:04.407 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:04.407 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:04.426 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:04.426 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:04.427 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:04.427 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.005 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.005 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.005 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.005 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.007 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.007 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.007 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.007 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.666 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.666 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.666 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.666 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.668 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.668 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.669 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:07.669 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.073 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.073 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.074 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.074 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.075 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.076 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.076 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.076 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.497 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.498 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.498 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.498 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.500 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.500 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.500 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.500 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.899 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.899 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.899 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.900 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.901 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.902 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.902 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:08.902 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.290 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.291 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.291 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.291 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.293 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.293 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.293 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.293 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.673 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.673 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.673 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.673 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.675 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.675 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.675 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:09.675 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.940 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.940 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.940 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.940 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.948 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.948 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.948 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.948 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.955 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.956 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.956 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.956 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.964 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.964 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.964 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:12.964 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.590 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.590 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.590 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.590 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.659 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.659 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.659 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.659 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.745 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.746 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.746 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.746 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.749 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.749 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.749 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.749 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.832 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.833 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.833 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.833 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.836 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.836 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.837 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.837 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.919 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.920 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.920 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.920 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.923 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.923 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.923 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:14.924 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.009 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.010 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.010 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.010 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.014 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.014 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.014 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.014 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.104 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.105 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.105 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.105 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.108 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.109 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.109 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:15.109 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.068 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.069 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.069 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.069 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.205 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.205 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.205 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.205 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.209 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.210 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.210 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.210 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.214 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.214 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.214 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.214 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.237 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.237 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.237 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.237 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.244 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.244 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.244 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:26.244 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.421 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.422 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.422 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.422 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.425 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.426 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.426 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.426 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.430 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.430 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.430 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.430 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.431 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.432 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.432 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:34.432 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:39.188 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:39.189 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:39.189 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:39.189 WARNING:scrape:Saw end of BT but not start
2024-03-02T09:25:53.051 INFO:scrape:Found 17 distinct failure reasons
2024-03-02T09:25:53.051 INFO:scrape:Failure: "2024-03-01T20:43:53.624540+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log
2024-03-02T09:25:53.051 INFO:scrape:45 jobs
2024-03-02T09:25:53.053 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2024-03-02T09:25:53.053 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-inline/no', '3-snaps/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'fail_fs/no', 'frag', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2024-03-02T09:25:53.053 INFO:scrape:
2024-03-02T09:25:53.053 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2024-03-02T09:25:53.053 INFO:scrape:1 jobs: ['7577856']
2024-03-02T09:25:53.053 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/pool-perm}']
2024-03-02T09:25:53.053 INFO:scrape:
2024-03-02T09:25:53.053 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi103.front.sepia.ceph.com: ['type=AVC msg=audit(1709317184.002:210): avc:  denied  { checkpoint_restore } for  pid=1202 comm="agetty" capability=40  scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1']
2024-03-02T09:25:53.054 INFO:scrape:11 jobs: ['7577689', '7577845', '7577633', '7577694', '7577805', '7577819', '7577839', '7577784', '7577761', '7577812', '7577664']
2024-03-02T09:25:53.054 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2024-03-02T09:25:53.054 INFO:scrape:suites union: ['1-ceph', '1-ranks/2', '1-tests/fscrypt-ffsb}}', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '2-logrotate}', '2-workunit/fs/snaps}}', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'bluestore-bitmap', 'centos_9.stream', 'centos_latest', 'clusters/1-mds-1-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'conf/{client', 'distro/{centos_latest}', 'fail_fs/no', 'frag', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'roles', 'session_timeout', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'syntax}', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/failover}', 'tasks/metrics}', 'tasks/multifs-auth}', 'tasks/trim-i24137}', 'tasks/truncate_delay}', 'tasks/volumes/{overrides', 'tasks/{0-client', 'tasks/{0-from/reef/{reef}', 'tasks/{1-thrash/mon', 'test/clone}}', 'thrashosds-health}', 'traceless/50pc}']
2024-03-02T09:25:53.054 INFO:scrape:
2024-03-02T09:25:53.054 INFO:scrape:Failure: "1709320200.0001485 mon.smithi003 (mon.0) 1002 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data" in cluster log
2024-03-02T09:25:53.054 INFO:scrape:2 jobs: ['7577707', '7577654']
2024-03-02T09:25:53.054 INFO:scrape:suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-03-02T09:25:53.054 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-03-02T09:25:53.055 INFO:scrape:
2024-03-02T09:25:53.055 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-03-02T09:25:53.055 INFO:scrape:5 jobs: ['7577842', '7577679', '7577636', '7577786', '7577745']
2024-03-02T09:25:53.055 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}']
2024-03-02T09:25:53.055 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}']
2024-03-02T09:25:53.055 INFO:scrape:
2024-03-02T09:25:53.055 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d5b56dd35348c337c1a3d5cf71dab209c79cf1a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2024-03-02T09:25:53.055 INFO:scrape:2 jobs: ['7577696', '7577747']
2024-03-02T09:25:53.055 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'n/3', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay']
2024-03-02T09:25:53.056 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}', 'wsync/yes}']
2024-03-02T09:25:53.056 INFO:scrape:
2024-03-02T09:25:53.056 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2024-03-02T09:25:53.056 INFO:scrape:5 jobs: ['7577785', '7577681', '7577693', '7577724', '7577626']
2024-03-02T09:25:53.056 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay']
2024-03-02T09:25:53.056 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2024-03-02T09:25:53.056 INFO:scrape:
2024-03-02T09:25:53.056 INFO:scrape:Failure: valgrind error: Leak_StillReachable
malloc
malloc
strdup
2024-03-02T09:25:53.057 INFO:scrape:3 jobs: ['7577754', '7577659', '7577798']
2024-03-02T09:25:53.057 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'validater/valgrind}']
2024-03-02T09:25:53.057 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}']
2024-03-02T09:25:53.057 INFO:scrape:
2024-03-02T09:25:53.057 INFO:scrape:Failure: Failed to reconnect to smithi121
2024-03-02T09:25:53.057 INFO:scrape:1 jobs: ['7577646']
2024-03-02T09:25:53.057 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{centos_9.stream', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2024-03-02T09:25:53.057 INFO:scrape:
2024-03-02T09:25:53.057 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)
2024-03-02T09:25:53.057 INFO:scrape:1 jobs: ['7577691']
2024-03-02T09:25:53.057 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}']
2024-03-02T09:25:53.058 INFO:scrape:
2024-03-02T09:25:53.058 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d5b56dd35348c337c1a3d5cf71dab209c79cf1a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2024-03-02T09:25:53.058 INFO:scrape:2 jobs: ['7577800', '7577644']
2024-03-02T09:25:53.058 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'n/5', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay']
2024-03-02T09:25:53.058 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}', 'wsync/yes}']
2024-03-02T09:25:53.058 INFO:scrape:
2024-03-02T09:25:53.058 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi029 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d5b56dd35348c337c1a3d5cf71dab209c79cf1a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'
2024-03-02T09:25:53.058 INFO:scrape:4 jobs: ['7577655', '7577762', '7577823', '7577711']
2024-03-02T09:25:53.058 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay']
2024-03-02T09:25:53.059 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{balancer/automatic', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2024-03-02T09:25:53.059 INFO:scrape:
2024-03-02T09:25:53.059 INFO:scrape:Failure: Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2024-03-02T09:25:53.059 INFO:scrape:2 jobs: ['7577809', '7577818']
2024-03-02T09:25:53.059 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse']
2024-03-02T09:25:53.059 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/mirror}', 'tasks/mirror}}']
2024-03-02T09:25:53.059 INFO:scrape:
2024-03-02T09:25:53.059 INFO:scrape:Failure: Command failed on smithi121 with status 1: 'sudo yum install -y kernel'
2024-03-02T09:25:53.059 INFO:scrape:1 jobs: ['7577645']
2024-03-02T09:25:53.060 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/fs/norstats}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2024-03-02T09:25:53.060 INFO:scrape:
2024-03-02T09:25:53.060 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts)
2024-03-02T09:25:53.060 INFO:scrape:1 jobs: ['7577815']
2024-03-02T09:25:53.060 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}']
2024-03-02T09:25:53.060 INFO:scrape:
2024-03-02T09:25:53.060 INFO:scrape:Dead: b'2024-03-02T09:13:44.559 DEBUG:teuthology.exit:Finished running handlers'
2024-03-02T09:25:53.060 INFO:scrape:2 jobs: ['7577847', '7577706']
2024-03-02T09:25:53.060 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2024-03-02T09:25:53.061 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'export-check', 'frag', 'fs/functional/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'n/3', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-no-extra-options}']
2024-03-02T09:25:53.061 INFO:scrape:
2024-03-02T09:25:53.061 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
2024-03-02T09:25:53.061 INFO:scrape:1 jobs: ['7577728']
2024-03-02T09:25:53.061 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client']
2024-03-02T09:25:53.061 INFO:scrape: