2022-09-26T16:36:01.547 INFO:root:teuthology version: 1.1.1.dev424+ga773b061
2022-09-26T16:36:01.549 INFO:teuthology.results:subset: '113/128'
2022-09-26T16:36:01.549 INFO:teuthology.results:seed: '3389'
2022-09-26T17:15:04.154 INFO:root:teuthology version: 1.1.1.dev424+ga773b061
2022-09-26T17:15:04.160 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-09-27T05:15:54.892 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2022-09-27T05:16:01.709 INFO:scrape:Found 42 jobs
2022-09-27T05:16:43.019 INFO:scrape:Found 20 distinct failure reasons
2022-09-27T05:16:43.021 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2022-09-27T05:16:43.021 INFO:scrape:7 jobs: ['7044446', '7044433', '7044459', '7044441', '7044454', '7044473', '7044451']
2022-09-27T05:16:43.022 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay']
2022-09-27T05:16:43.023 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2022-09-27T05:16:43.023 INFO:scrape:
2022-09-27T05:16:43.024 INFO:scrape:Failure: Test failure: test_dump_loads (tasks.cephfs.test_admin.TestAdminCommandDumpLoads)
2022-09-27T05:16:43.024 INFO:scrape:1 jobs: ['7044453']
2022-09-27T05:16:43.025 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/admin}']
2022-09-27T05:16:43.025 INFO:scrape:
2022-09-27T05:16:43.026 INFO:scrape:Failure: Command failed on smithi169 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib'
2022-09-27T05:16:43.026 INFO:scrape:1 jobs: ['7044438']
2022-09-27T05:16:43.027 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}']
2022-09-27T05:16:43.027 INFO:scrape:
2022-09-27T05:16:43.028 INFO:scrape:Failure: Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
2022-09-27T05:16:43.028 INFO:scrape:2 jobs: ['7044447', '7044470']
2022-09-27T05:16:43.029 INFO:scrape:suites intersection: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus']
2022-09-27T05:16:43.030 INFO:scrape:suites union: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus']
2022-09-27T05:16:43.031 INFO:scrape:
2022-09-27T05:16:43.032 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2022-09-27T05:16:43.033 INFO:scrape:1 jobs: ['7044466']
2022-09-27T05:16:43.034 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}']
2022-09-27T05:16:43.035 INFO:scrape:
2022-09-27T05:16:43.036 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds
2022-09-27T05:16:43.037 INFO:scrape:8 jobs: ['7044457', '7044449', '7044452', '7044448', '7044439', '7044434', '7044460', '7044440']
2022-09-27T05:16:43.038 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay']
2022-09-27T05:16:43.039 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2022-09-27T05:16:43.040 INFO:scrape:
2022-09-27T05:16:43.041 INFO:scrape:Failure: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
2022-09-27T05:16:43.041 INFO:scrape:1 jobs: ['7044462']
2022-09-27T05:16:43.042 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}']
2022-09-27T05:16:43.044 INFO:scrape:
2022-09-27T05:16:43.045 INFO:scrape:Failure: Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
2022-09-27T05:16:43.046 INFO:scrape:1 jobs: ['7044458']
2022-09-27T05:16:43.047 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}']
2022-09-27T05:16:43.048 INFO:scrape:
2022-09-27T05:16:43.049 INFO:scrape:Failure: "1664219769.8010643 osd.1 (osd.1) 32 : cluster [ERR] 5.12s0 deep-scrub : stat mismatch, got 316/324 objects, 0/0 clones, 316/324 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 2519930/2539108 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
2022-09-27T05:16:43.049 INFO:scrape:1 jobs: ['7044444']
2022-09-27T05:16:43.050 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-09-27T05:16:43.051 INFO:scrape:
2022-09-27T05:16:43.053 INFO:scrape:Failure: "1664213910.14213 mds.f (mds.0) 15 : cluster [WRN] client.4717 isn't responding to mclientcaps(revoke), ino 0x100000098e8 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 304.960864 seconds ago" in cluster log
2022-09-27T05:16:43.054 INFO:scrape:1 jobs: ['7044463']
2022-09-27T05:16:43.055 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-09-27T05:16:43.055 INFO:scrape:
2022-09-27T05:16:43.056 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2022-09-27T05:16:43.057 INFO:scrape:2 jobs: ['7044468', '7044450']
2022-09-27T05:16:43.058 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'wsync/yes}']
2022-09-27T05:16:43.059 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2022-09-27T05:16:43.060 INFO:scrape:
2022-09-27T05:16:43.061 INFO:scrape:Failure: "1664211423.2612686 mon.a (mon.0) 351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022-09-27T05:16:43.062 INFO:scrape:2 jobs: ['7044445', '7044435']
2022-09-27T05:16:43.063 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health']
2022-09-27T05:16:43.064 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/metrics}', 'tasks/snap-schedule}']
2022-09-27T05:16:43.065 INFO:scrape:
2022-09-27T05:16:43.066 INFO:scrape:Dead: b'2022-09-27T05:08:58.504 DEBUG:teuthology.exit:Finished running handlers'
2022-09-27T05:16:43.067 INFO:scrape:1 jobs: ['7044467']
2022-09-27T05:16:43.068 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2022-09-27T05:16:43.069 INFO:scrape:
2022-09-27T05:16:43.069 INFO:scrape:Failure: "1664213240.635227 mds.a (mds.0) 4 : cluster [WRN] Scrub error on dir 0x10000000001 (/parent/flushed) see mds.a log and `damage ls` output for details" in cluster log
2022-09-27T05:16:43.071 INFO:scrape:1 jobs: ['7044469']
2022-09-27T05:16:43.072 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/forward-scrub}']
2022-09-27T05:16:43.073 INFO:scrape:
2022-09-27T05:16:43.074 INFO:scrape:Dead: b"2022-09-26T17:24:08.387 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044464', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-26 16:57:34.275230', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDARs7gTETc9F/f4Wi4kaMZ64MKjozrXEJlud374Tc3ITq5Jl+PWIwMwg1mzXFm+0MjAycfjaRsictClOSRh27RIKiU/b0ciEOIGjH0H+lKkeqZBnxs5UtGF/SQYPK1oj6/pDwB8g0lcd75pL90w3paxY7ENQrJL6lbEWhF3Tix4RKVrGr9+V2hUdE6xrMDiQpswa7A/r9mW2ONt2KQ6A/rGpy7B6MSnRrpdc/2Le/LYamnjOxSZVvwRK7P8ChqugaXsgJHANResAGdPn6cJhKfj3YUBbiKSRh6bHjpzmc8KTcOztTOAdCLp+d5ccLfocK7e32yoEoTXB1K9lmL0qP4Qdaa5u7o6EIcqy2uH5s7bnfYK5WLxx3nN6OO1muev4OcPCCZnrwOB60JNsHbiNadXxkv637Iiec2OcUQqCUtKGMDnh0CblNOVro4678su+0ekXBnC6W2RvffXy6xwzdQC+D4dyWFXENTs/tsdAaa6dGX4eIYBjBVvp8aD4JWkJM='}"
2022-09-27T05:16:43.074 INFO:scrape:1 jobs: ['7044464']
2022-09-27T05:16:43.075 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}']
2022-09-27T05:16:43.076 INFO:scrape:
2022-09-27T05:16:43.079 INFO:scrape:Failure: Stale jobs detected, aborting.
2022-09-27T05:16:43.080 INFO:scrape:1 jobs: ['7044471']
2022-09-27T05:16:43.081 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2022-09-27T05:16:43.082 INFO:scrape:
2022-09-27T05:16:43.083 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2022-09-27T05:16:43.084 INFO:scrape:2 jobs: ['7044461', '7044443']
2022-09-27T05:16:43.085 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'wsync/yes}']
2022-09-27T05:16:43.086 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}']
2022-09-27T05:16:43.087 INFO:scrape:
2022-09-27T05:16:43.088 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing
2022-09-27T05:16:43.089 INFO:scrape:2 jobs: ['7044455', '7044472']
2022-09-27T05:16:43.090 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2022-09-27T05:16:43.091 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2022-09-27T05:16:43.092 INFO:scrape:
2022-09-27T05:16:43.092 INFO:scrape:Failure: {'smithi169.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi169.front.sepia.ceph.com,172.21.15.169' (ECDSA) to the list of known hosts.\r\nubuntu@smithi169.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive)."}}
2022-09-27T05:16:43.093 INFO:scrape:1 jobs: ['7044437']
2022-09-27T05:16:43.093 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}']
2022-09-27T05:16:43.094 INFO:scrape:
2022-09-27T05:16:43.095 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi125 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'
2022-09-27T05:16:43.096 INFO:scrape:1 jobs: ['7044465']
2022-09-27T05:16:43.096 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}']
2022-09-27T05:16:43.097 INFO:scrape:
2022-09-27T05:16:43.100 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/
failed:  36
dead:    1
running: 1
waiting: 0
queued:  0
passed:  4


Fail
=================================================================
[7044433]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:31:45
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044433/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044433/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2cb0fd528dc2472a8fb835d0ad23acd4

    Command failed (workunit test suites/pjd.sh) on smithi059 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7044434]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
-----------------------------------------------------------------
time:   02:15:22
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044434/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044434/

    error during scrub thrashing: reached maximum tries (30) after waiting for
    900 seconds

[7044435]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:42:25
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044435/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044435/

    "1664212949.7832673 mon.a (mon.0) 1705 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)"
    in cluster log

[7044437]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:12:08
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044437/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044437/
sentry: https://sentry.ceph.com/organizations/ceph/?query=87e9b06521cb4d16b2f3309cc349c701

    {'smithi169.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg':
    "Failed to connect to the host via ssh: Warning: Permanently added
    'smithi169.front.sepia.ceph.com,172.21.15.169' (ECDSA) to the list of known
    hosts.\r\nubuntu@smithi169.front.sepia.ceph.com: Permission denied
    (publickey,password,keyboard-interactive)."}}

[7044438]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:11:38
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044438/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044438/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c32a98381aef45db8996a237a48f7188

    Command failed on smithi169 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib'

[7044439]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:43:29
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044439/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044439/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7044440]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   01:29:18
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044440/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044440/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7044441]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:40:28
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044441/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044441/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c465f27a59f04bb0b4ee92ddd55938d4

    Command failed (workunit test suites/pjd.sh) on smithi063 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7044443]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:43:53
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044443/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044443/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ffc33cbc3b2e485f8bd7328bb6e28971

    Command failed (workunit test suites/ffsb.sh) on smithi033 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

[7044444]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   02:55:15
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044444/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044444/

    "1664219769.8010643 osd.1 (osd.1) 32 : cluster [ERR] 5.12s0 deep-scrub :
    stat mismatch, got 316/324 objects, 0/0 clones, 316/324 dirty, 0/0 omap,
    0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 2519930/2539108 bytes, 0/0
    manifest objects, 0/0 hit_set_archive bytes." in cluster log

[7044445]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:50:05
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044445/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044445/

    "1664211423.2612686 mon.a (mon.0) 351 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7044446]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:42:43
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044446/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044446/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a79d746dcc85416197d81d60080af48c

    Command failed (workunit test suites/pjd.sh) on smithi032 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7044447]  fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
-----------------------------------------------------------------
time:   00:11:45
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044447/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044447/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a461a6b6c1e8426a9f4619cc854d519f

    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)

[7044448]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   02:18:09
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044448/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044448/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7044449]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:56:28
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044449/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044449/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7044450]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   01:01:58
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044450/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044450/
sentry: https://sentry.ceph.com/organizations/ceph/?query=087e15dd7d4e40de97c4682d24ba9651

    Command failed (workunit test suites/dbench.sh) on smithi005 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7044451]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:33:02
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044451/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044451/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d544a2dca0b4c5499e34ff6d5f188aa

    Command failed (workunit test suites/pjd.sh) on smithi052 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7044452]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:56:39
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044452/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044452/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7044453]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:18:59
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044453/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044453/
sentry: https://sentry.ceph.com/organizations/ceph/?query=904f83e917b449f382eac3676e9c29b6

    Test failure: test_dump_loads
    (tasks.cephfs.test_admin.TestAdminCommandDumpLoads)

[7044454]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:35:12
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044454/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044454/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ea35056ca5574feca6ed06b92a7b0bd6

    Command failed (workunit test suites/pjd.sh) on smithi006 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7044455]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044455/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044455/
sentry: https://sentry.ceph.com/organizations/ceph/?query=de697d93cbb64d79afeebe04b40b3750

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&re
    f=testing

[7044457]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}}
-----------------------------------------------------------------
time:   01:11:42
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044457/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044457/

    error during scrub thrashing: reached maximum tries (30) after waiting for
    900 seconds

[7044458]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:34:45
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044458/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044458/
sentry: https://sentry.ceph.com/organizations/ceph/?query=58ce7581d7c44fd39966892479fe5ad9

    Test failure: test_subvolume_group_ls_filter_internal_directories
    (tasks.cephfs.test_volumes.TestSubvolumeGroups)

[7044459]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:36:46
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044459/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044459/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ee3307be77f94926b661242b94be8c2f

    Command failed (workunit test suites/pjd.sh) on smithi036 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7044460]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   01:30:56
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044460/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044460/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7044461]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:56:09
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044461/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044461/
sentry: https://sentry.ceph.com/organizations/ceph/?query=05d101db523b40a1a468745f022efa34

    Command failed (workunit test suites/ffsb.sh) on smithi055 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

[7044462]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:22:59
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044462/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044462/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dee2381e79c04a77ab16da882ca59cdf

    Test failure: test_reconnect_after_blocklisted
    (tasks.cephfs.test_client_recovery.TestClientRecovery)

[7044463]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   01:56:21
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044463/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044463/

    "1664213910.14213 mds.f (mds.0) 15 : cluster [WRN] client.4717 isn't
    responding to mclientcaps(revoke), ino 0x100000098e8 pending pAsLsXsFscr
    issued pAsLsXsFsxcrwb, sent 304.960864 seconds ago" in cluster log

[7044465]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
-----------------------------------------------------------------
time:   00:13:42
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044465/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044465/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d2e7a8ad94764335a308ea4fa74547ac

    Command failed (workunit test libcephfs/test.sh) on smithi125 with status
    139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'

[7044466]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:49:05
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044466/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044466/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6dd8c0031b2b4db68d008dd02400e00e

    Test failure: test_subvolume_snapshot_info_if_orphan_clone
    (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)

[7044468]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:56:50
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044468/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044468/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ec57c5e806e045e59b982741eb8cf0fc

    Command failed (workunit test suites/dbench.sh) on smithi079 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7044469]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:17:23
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044469/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044469/

    "1664213240.635227 mds.a (mds.0) 4 : cluster [WRN] Scrub error on dir
    0x10000000001 (/parent/flushed) see mds.a log and `damage ls` output for
    details" in cluster log

[7044470]  fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
-----------------------------------------------------------------
time:   00:10:40
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044470/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044470/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2c272dfc9f50448c81f2f425d30e4fd4

    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)

[7044471]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044471/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044471/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2421249799a74e0388d0cfcd98b75e88

    Stale jobs detected, aborting.

[7044472]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044472/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044472/
sentry: https://sentry.ceph.com/organizations/ceph/?query=02c35a026cf24188a712d602e4f1775b

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&re
    f=testing

[7044473]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:32:52
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044473/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044473/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7f899155a5244973bb51598530e6ac2c

    Command failed (workunit test suites/pjd.sh) on smithi096 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=21fa21e37e88ed873d641d9e5c90110b817d733d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'



Dead
=================================================================
[7044464]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044464/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044464/



Running
=================================================================
[7044467] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}}
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044467/



Pass
=================================================================
[7044436] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}}
time:   00:47:23
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044436/

[7044442] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:25:42
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044442/

[7044456] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}}
time:   02:10:16
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044456/

[7044474] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
time:   00:42:00
info:   http://pulpito.ceph.com/pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044474/