2022-07-01T00:21:18.533 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-01T00:21:18.534 INFO:teuthology.results:subset: '18/32' 2022-07-01T00:21:18.535 INFO:teuthology.results:seed: '2741' 2022-07-04T09:12:32.615 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-04T09:12:32.653 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-04T11:10:44.271 INFO:teuthology.results:Tests finished! gathering results... 2022-07-04T11:11:16.298 INFO:scrape:Found 292 jobs 2022-07-04T11:11:18.415 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902032/remote/smithi109/log/ceph-mgr.y.log.gz 2022-07-04T11:11:18.418 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:36.600 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:36.602 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:36.789 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:36.790 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:36.960 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:36.962 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:37.226 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:37.228 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:37.429 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:37.430 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:37.656 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:37.657 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:37.818 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:37.819 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:38.008 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:38.009 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:38.200 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:38.201 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:38.462 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:38.464 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:38.698 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:38.699 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:38.919 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:38.921 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:39.111 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:39.113 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:39.336 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:39.337 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:39.540 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:39.541 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:39.709 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/remote/smithi042/log/ceph-mds.a.log.gz 2022-07-04T11:11:39.711 WARNING:scrape:Not-understood crash indication 2022-07-04T11:11:44.403 INFO:scrape:Found 17 distinct failure reasons 2022-07-04T11:11:44.405 INFO:scrape:Failure: SSH connection to smithi083 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2022-07-04T11:11:44.405 INFO:scrape:7 jobs: ['6902075', '6901971', '6902048', '6901951', '6902005', '6901998', '6902095'] 2022-07-04T11:11:44.406 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount', 'osd}', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.406 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{mon-debug', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/fsstress', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'validater/lockdep}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/pjd}', 'wsync/{yes}}'] 2022-07-04T11:11:44.407 INFO:scrape: 2022-07-04T11:11:44.407 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-07-04T11:11:44.408 INFO:scrape:4 jobs: ['6901884', '6902034', '6902084', '6901894'] 2022-07-04T11:11:44.408 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.409 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.409 INFO:scrape: 2022-07-04T11:11:44.410 INFO:scrape:Crash: "2022-07-04T09:25:46.841103+0000 mon.a (mon.0) 1371 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log 2022-07-04T11:11:44.411 INFO:scrape:ceph version 17.2.0-501-gffefd1c0 (ffefd1c03754977bf5b4bbc82ce524944f0e0c52) quincy (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f571b21bb20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f571c237c32] 5: /usr/lib64/ceph/libceph-common.so.2(+0x283df5) [0x7f571c237df5] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x12b0) [0x556fce1818d0] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xfed) [0x556fcdead11d] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x7cb) [0x556fcdeafabb] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556fcdeb00dc] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x108) [0x556fcde9efd8] 11: (DispatchQueue::entry()+0x14fa) [0x7f571c4be43a] 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f571c575581] 13: /lib64/libpthread.so.0(+0x814a) [0x7f571b21114a] 14: clone() 2022-07-04T11:11:44.412 INFO:scrape:1 jobs: ['6902125'] 2022-07-04T11:11:44.412 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.413 INFO:scrape: 2022-07-04T11:11:44.414 INFO:scrape:Dead: b'2022-07-01T15:45:58.717 DEBUG:teuthology.exit:Finished running handlers' 2022-07-04T11:11:44.415 INFO:scrape:15 jobs: ['6902032', '6901852', '6901937', '6902049', '6902001', '6902025', '6902013', '6901860', '6902057', '6901939', '6901980', '6902058', '6902102', '6902040', '6902067'] 2022-07-04T11:11:44.415 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.416 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{mon-debug', 'overrides/{no_client_pidfile', 'overrides/{pg-warn', 'overrides}}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'roles', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'syntax', 'tasks/data-scan}', 'tasks/fsstress', 'tasks/{0-check-counter', 'tasks/{0-from/v16.2.4', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'validater/lockdep}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/kernel_untar_build}', 'workunit/suites/blogbench}', 'workunit/suites/dbench}', 'workunit/suites/fsstress}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-07-04T11:11:44.416 INFO:scrape: 2022-07-04T11:11:44.417 INFO:scrape:Failure: {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} 2022-07-04T11:11:44.417 INFO:scrape:7 jobs: ['6901921', '6901880', '6902093', '6902100', '6902105', '6901832', '6902076'] 2022-07-04T11:11:44.418 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.418 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/cfuse_workunit_suites_pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{rhel_8}', 'fs/32bits/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{whitelist_health', 'ranks/1', 'ranks/5', 'rhel_8}', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/workunit/snaps}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/norstats}', 'workunit/suites/ffsb}', 'workunit/suites/fsync-tester}', 'workunit/suites/iogen}', 'wsync/{yes}}'] 2022-07-04T11:11:44.419 INFO:scrape: 2022-07-04T11:11:44.420 INFO:scrape:Failure: Command failed on smithi162 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2022-07-04T11:11:44.420 INFO:scrape:6 jobs: ['6902059', '6901758', '6901917', '6902091', '6901789', '6901794'] 2022-07-04T11:11:44.421 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.421 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-client', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-upgrade', '2-workunit/suites/iozone}}', '3-compat_client/no}}', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{secure}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{multimds/yes', 'overrides/{pg-warn', 'pg-warn', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'roles', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'syntax', 'tasks/{0-check-counter', 'tasks/{0-from/pacific', 'tasks/{0-octopus', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/fs/test_o_trunc}', 'workunit/suites/fsync-tester}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-07-04T11:11:44.422 INFO:scrape: 2022-07-04T11:11:44.422 INFO:scrape:Failure: "2022-07-04T09:30:22.414806+0000 mon.a (mon.0) 1521 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log 2022-07-04T11:11:44.423 INFO:scrape:2 jobs: ['6902119', '6901788'] 2022-07-04T11:11:44.423 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'osd}', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.424 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.424 INFO:scrape: 2022-07-04T11:11:44.425 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2022-07-04T11:11:44.425 INFO:scrape:1 jobs: ['6901994'] 2022-07-04T11:11:44.426 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{whitelist_health', 'tasks/workunit/snaps}', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.426 INFO:scrape: 2022-07-04T11:11:44.427 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) 2022-07-04T11:11:44.427 INFO:scrape:1 jobs: ['6901965'] 2022-07-04T11:11:44.427 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/client-readahead}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.428 INFO:scrape: 2022-07-04T11:11:44.428 INFO:scrape:Crash: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' 2022-07-04T11:11:44.429 INFO:scrape:ceph version 17.2.0-501-gffefd1c0 (ffefd1c03754977bf5b4bbc82ce524944f0e0c52) quincy (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f55bd41cce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f55bdfe9c32] 5: /usr/lib64/ceph/libceph-common.so.2(+0x283df5) [0x7f55bdfe9df5] 6: (std::_Sp_counted_ptr::_M_dispose()+0x1cd) [0x5643753890dd] 7: (std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x47) [0x5643753860d7] 8: (Client::handle_client_session(boost::intrusive_ptr const&)+0x237) [0x56437536d017] 9: (Client::ms_dispatch2(boost::intrusive_ptr const&)+0x22d) [0x56437537101d] 10: (DispatchQueue::entry()+0x14fa) [0x7f55be27043a] 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55be327581] 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55bd4121ca] 13: clone() 2022-07-04T11:11:44.429 INFO:scrape:1 jobs: ['6901759'] 2022-07-04T11:11:44.430 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.430 INFO:scrape: 2022-07-04T11:11:44.431 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-snap-rm-cmp.sh) on smithi124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap-rm-cmp.sh' 2022-07-04T11:11:44.431 INFO:scrape:1 jobs: ['6901872'] 2022-07-04T11:11:44.432 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.432 INFO:scrape: 2022-07-04T11:11:44.433 INFO:scrape:Failure: SSH connection to smithi107 was lost: 'sudo PATH=/usr/sbin:$PATH lsof ; ps auxf' 2022-07-04T11:11:44.433 INFO:scrape:1 jobs: ['6902070'] 2022-07-04T11:11:44.433 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{osd-asserts', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.434 INFO:scrape: 2022-07-04T11:11:44.434 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_clone_with_upgrade (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones), test_subvolume_snapshot_clone_with_upgrade (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2022-07-04T11:11:44.435 INFO:scrape:1 jobs: ['6902103'] 2022-07-04T11:11:44.435 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.436 INFO:scrape: 2022-07-04T11:11:44.436 INFO:scrape:Failure: error during scrub thrashing: Command failed on smithi070 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' 2022-07-04T11:11:44.437 INFO:scrape:1 jobs: ['6901774'] 2022-07-04T11:11:44.437 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}', 'wsync/{yes}}'] 2022-07-04T11:11:44.437 INFO:scrape: 2022-07-04T11:11:44.438 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi042 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2022-07-04T11:11:44.438 INFO:scrape:1 jobs: ['6902085'] 2022-07-04T11:11:44.439 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.439 INFO:scrape: 2022-07-04T11:11:44.440 INFO:scrape:Failure: {'smithi099.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi161.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi066.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} 2022-07-04T11:11:44.440 INFO:scrape:1 jobs: ['6902097'] 2022-07-04T11:11:44.441 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/test_o_trunc}', 'wsync/{yes}}'] 2022-07-04T11:11:44.441 INFO:scrape: 2022-07-04T11:11:44.442 INFO:scrape:Failure: Test failure: test_scrub_pause_and_resume_with_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) 2022-07-04T11:11:44.442 INFO:scrape:1 jobs: ['6902107'] 2022-07-04T11:11:44.443 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-04T11:11:44.443 INFO:scrape: 2022-07-04T11:11:44.461 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/ failed: 29 dead: 23 running: 0 waiting: 0 queued: 0 passed: 240 Fail ================================================================= [6901758] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} ----------------------------------------------------------------- time: 00:29:05 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901758/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901758/ Command failed on smithi026 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [6901759] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:22:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901759/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901759/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d3aad8e4c4f4094a8e6ba6a90970e7e Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6901774] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 01:10:17 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901774/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901774/ error during scrub thrashing: Command failed on smithi070 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6901788] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:53:13 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901788/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901788/ "2022-07-01T01:53:29.118848+0000 mon.a (mon.0) 442 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6901789] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:14:12 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901789/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901789/ Command failed on smithi026 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [6901794] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} ----------------------------------------------------------------- time: 00:34:01 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901794/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901794/ Command failed on smithi162 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [6901872] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:39:45 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901872/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901872/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba33c9dbe4c94599aaa0d5f3ac69b786 Command failed (workunit test fs/snaps/snaptest-snap-rm-cmp.sh) on smithi124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap- rm-cmp.sh' [6901884] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:23:06 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901884/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901884/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdaadbc7d86244fa83925cd465c84841 reached maximum tries (90) after waiting for 540 seconds [6901894] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:25:18 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901894/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e76497455d4443509df0df3f3b204027 reached maximum tries (90) after waiting for 540 seconds [6901917] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} ----------------------------------------------------------------- time: 00:31:20 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901917/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901917/ Command failed on smithi080 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [6901951] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:31:53 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901951/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901951/ sentry: https://sentry.ceph.com/organizations/ceph/?query=431a567d591a4955ae85445cd829bbf2 SSH connection to smithi083 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6901965] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} ----------------------------------------------------------------- time: 00:31:37 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901965/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901965/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63add266952b40d1a0b40236fb4d4a1e Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) [6901971] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:36:07 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901971/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901971/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5312c94249b049c2bf648dffd595755a SSH connection to smithi143 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6901994] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:23:49 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901994/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901994/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db10b5791db745b789ef2057681096c1 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [6901998] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:32:52 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901998/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901998/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3712832b8b9b4d4997d2a660a0c1b86f SSH connection to smithi003 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6902005] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:31:09 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902005/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902005/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70f5b4ff85664bf0ad987faf847b2a7e Command crashed: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6902034] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:20:02 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902034/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902034/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2e6571b2a734e24855159dee4956fd8 reached maximum tries (90) after waiting for 540 seconds [6902048] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} ----------------------------------------------------------------- time: 00:41:36 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902048/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902048/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cfa6a8a5943848d6bd2c3fa7a7bae5cc SSH connection to smithi060 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6902059] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:31:34 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902059/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902059/ Command failed on smithi162 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [6902070] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 11:33:20 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902070/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902070/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a52392070ea4063b1246497ce6b6a09 SSH connection to smithi107 was lost: 'sudo PATH=/usr/sbin:$PATH lsof ; ps auxf' [6902075] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:29:10 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902075/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902075/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d91c23cbd79c4fbe91c3c88aeca1b535 SSH connection to smithi083 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6902084] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:19:50 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902084/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902084/ sentry: https://sentry.ceph.com/organizations/ceph/?query=82b80b20c93340d29de036882092f0e7 reached maximum tries (90) after waiting for 540 seconds [6902085] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:39:43 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902085/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8c577fc4cb34863b63cd6e9b17ce258 Command failed (workunit test suites/fsstress.sh) on smithi042 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ffefd1c03754977bf5b4bbc82ce524944f0e0c52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [6902091] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:28:26 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902091/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902091/ Command failed on smithi162 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [6902095] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:32:08 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902095/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902095/ sentry: https://sentry.ceph.com/organizations/ceph/?query=755c7c465cd34461b43c38f1ad175153 SSH connection to smithi170 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6902103] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 01:07:38 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902103/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902103/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6763423099f64adf8cf31161a5c40a7e Test failure: test_subvolume_snapshot_clone_with_upgrade (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones), test_subvolume_snapshot_clone_with_upgrade (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [6902107] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} ----------------------------------------------------------------- time: 00:12:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902107/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902107/ sentry: https://sentry.ceph.com/organizations/ceph/?query=437429415c1a415aa5eaaf0237db6a4e Test failure: test_scrub_pause_and_resume_with_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) [6902119] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:42:56 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902119/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902119/ "2022-07-04T09:30:22.414806+0000 mon.a (mon.0) 1521 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [6902125] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} ----------------------------------------------------------------- time: 00:18:42 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902125/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902125/ "2022-07-04T09:25:46.841103+0000 mon.a (mon.0) 1371 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log Dead ================================================================= [6901832] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:07:29 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901832/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901832/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71385558a3ae4071945089f059d1d75d {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6901852] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901852/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901852/ hit max job timeout [6901860] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901860/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901860/ hit max job timeout [6901880] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} ----------------------------------------------------------------- time: 00:08:59 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901880/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901880/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c9a3719479f475a9fd5403ad3211008 {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6901921] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:07:31 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901921/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901921/ sentry: https://sentry.ceph.com/organizations/ceph/?query=279a7bec574c4f09b90664cfecb04c0e {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6901937] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901937/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901937/ hit max job timeout [6901939] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901939/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901939/ hit max job timeout [6901980] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901980/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901980/ hit max job timeout [6902001] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902001/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902001/ hit max job timeout [6902013] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902013/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902013/ hit max job timeout [6902025] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902025/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902025/ hit max job timeout [6902032] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902032/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902032/ hit max job timeout [6902040] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902040/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902040/ hit max job timeout [6902049] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902049/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902049/ hit max job timeout [6902057] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902057/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902057/ hit max job timeout [6902058] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902058/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902058/ hit max job timeout [6902067] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902067/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902067/ hit max job timeout [6902076] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 00:06:49 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902076/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902076/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abc4bca727ec4e5492c62a21daec76cd {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6902093] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} ----------------------------------------------------------------- time: 00:08:39 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902093/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902093/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e739d0414154904b3cb0bc776831d09 {'smithi130.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6902097] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} ----------------------------------------------------------------- time: 00:05:44 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902097/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902097/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f9f366314ae417396edf898f0762a88 {'smithi099.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi161.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi066.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6902100] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:08:32 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902100/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902100/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a06f4b3df9c04f3cb6d9696737ec1252 {'smithi179.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi086.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6902102] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902102/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902102/ hit max job timeout [6902105] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} ----------------------------------------------------------------- time: 00:08:06 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902105/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=09193816fa384a5d8981bf3e3b835b84 {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} Pass ================================================================= [6901726] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:42:43 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901726/ [6901727] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} time: 00:23:42 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901727/ [6901728] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 01:30:11 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901728/ [6901729] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:28:01 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901729/ [6901730] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:27:28 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901730/ [6901731] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:12:11 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901731/ [6901732] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:25:35 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901732/ [6901733] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} time: 00:18:37 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901733/ [6901734] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:16:54 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901734/ [6901735] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} time: 00:47:56 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901735/ [6901736] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:20:26 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901736/ [6901737] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} time: 00:19:30 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901737/ [6901738] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:30 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901738/ [6901739] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i24137} time: 00:13:10 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901739/ [6901740] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:46:31 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901740/ [6901741] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full} time: 00:15:34 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901741/ [6901742] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client} time: 00:12:03 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901742/ [6901743] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 01:55:39 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901743/ [6901744] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit} time: 00:22:50 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901744/ [6901745] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 01:12:19 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901745/ [6901746] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:48:09 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901746/ [6901747] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} time: 00:28:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901747/ [6901748] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:10 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901748/ [6901749] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell} time: 00:35:31 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901749/ [6901750] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:44:24 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901750/ [6901751] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:04:09 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901751/ [6901752] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{rhel_8} tasks/fstop} time: 00:11:08 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901752/ [6901753] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:23 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901753/ [6901754] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:38 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901754/ [6901755] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} time: 02:10:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901755/ [6901756] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/valgrind} time: 00:56:08 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901756/ [6901757] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 01:02:28 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901757/ [6901760] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 00:53:12 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901760/ [6901761] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} time: 00:11:12 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901761/ [6901762] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:44:54 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901762/ [6901763] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 00:48:32 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901763/ [6901764] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:29:25 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901764/ [6901765] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota} time: 00:12:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901765/ [6901766] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:58 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901766/ [6901767] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:12:21 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901767/ [6901768] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:34:51 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901768/ [6901769] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:15:01 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901769/ [6901770] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs} time: 00:16:31 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901770/ [6901771] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} time: 01:46:57 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901771/ [6901772] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:21:04 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901772/ [6901773] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:45:35 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901773/ [6901775] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:12:05 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901775/ [6901776] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} time: 00:19:35 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901776/ [6901777] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:39:02 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901777/ [6901778] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:25:38 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901778/ [6901779] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:25:46 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901779/ [6901780] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap} time: 00:13:53 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901780/ [6901781] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:22:36 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901781/ [6901782] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} time: 00:28:28 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901782/ [6901783] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:54 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901783/ [6901784] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:13:21 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901784/ [6901785] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:11 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901785/ [6901786] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:13:15 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901786/ [6901787] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:35:51 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901787/ [6901790] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:29:34 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901790/ [6901791] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:19:31 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901791/ [6901792] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} time: 00:28:53 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901792/ [6901793] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:39:06 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901793/ [6901795] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901795/ [6901796] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:11:57 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901796/ [6901797] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:50:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901797/ [6901798] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:44:30 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901798/ [6901800] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:55:27 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901800/ [6901802] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/lockdep} time: 00:25:38 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901802/ [6901804] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} time: 00:44:49 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901804/ [6901806] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} time: 00:28:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901806/ [6901808] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 00:47:47 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901808/ [6901810] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:32:15 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901810/ [6901812] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:29:04 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901812/ [6901814] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:25:04 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901814/ [6901816] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:13:41 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901816/ [6901818] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:26:44 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901818/ [6901820] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration} time: 00:24:52 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901820/ [6901822] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:18:06 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901822/ [6901824] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:26:06 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901824/ [6901826] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:28:35 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901826/ [6901828] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:20:23 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901828/ [6901830] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay} time: 00:11:49 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901830/ [6901834] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:31:09 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901834/ [6901836] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:11:35 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901836/ [6901838] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} time: 00:54:15 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901838/ [6901840] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:43 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901840/ [6901842] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/dir-max-entries} time: 03:10:55 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901842/ [6901844] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:22:44 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901844/ [6901846] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:41 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901846/ [6901848] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/ino_release_cb} time: 00:12:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901848/ [6901850] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:32 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901850/ [6901854] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:54:18 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901854/ [6901856] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota} time: 00:18:45 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901856/ [6901858] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:12:26 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901858/ [6901862] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:16:25 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901862/ [6901864] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:21 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901864/ [6901866] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} time: 00:12:54 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901866/ [6901868] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:16:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901868/ [6901870] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:20:49 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901870/ [6901874] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 00:41:04 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901874/ [6901876] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} time: 00:48:57 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901876/ [6901878] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:44:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901878/ [6901882] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:38 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901882/ [6901886] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} time: 00:19:19 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901886/ [6901888] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} time: 00:21:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901888/ [6901890] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} time: 00:32:12 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901890/ [6901892] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:10:14 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901892/ [6901896] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/valgrind} time: 00:52:47 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901896/ [6901899] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:23:43 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901899/ [6901901] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} time: 00:29:02 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901901/ [6901903] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:14:47 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901903/ [6901905] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:36:07 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901905/ [6901907] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} time: 00:23:07 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901907/ [6901909] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:16:58 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901909/ [6901911] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:39 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901911/ [6901913] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:14:25 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901913/ [6901915] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:44:25 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901915/ [6901919] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:21:36 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901919/ [6901923] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:04 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901923/ [6901925] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} time: 00:13:25 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901925/ [6901927] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:58:18 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901927/ [6901929] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:34:48 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901929/ [6901931] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:26:55 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901931/ [6901933] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 01:05:59 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901933/ [6901935] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} time: 00:14:44 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901935/ [6901941] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:13:08 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901941/ [6901943] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} time: 00:32:12 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901943/ [6901945] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush} time: 00:12:01 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901945/ [6901947] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:17:26 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901947/ [6901949] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:25:08 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901949/ [6901953] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:23:37 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901953/ [6901955] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} time: 00:18:56 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901955/ [6901957] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:34:46 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901957/ [6901959] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} time: 00:31:05 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901959/ [6901961] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:25:08 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901961/ [6901963] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 00:53:37 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901963/ [6901967] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:36:40 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901967/ [6901969] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 02:00:52 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901969/ [6901973] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:33 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901973/ [6901975] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} time: 00:34:31 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901975/ [6901978] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:12:51 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901978/ [6901981] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:12:44 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901981/ [6901984] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/whitelist_health} time: 00:19:18 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901984/ [6901986] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}} time: 00:13:59 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901986/ [6901988] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:48:51 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901988/ [6901990] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:50:17 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901990/ [6901992] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:13:33 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901992/ [6901996] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:16:19 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6901996/ [6902000] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:30:26 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902000/ [6902003] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} time: 00:19:50 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902003/ [6902007] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902007/ [6902009] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:18:31 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902009/ [6902011] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} time: 01:02:53 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902011/ [6902015] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:21:31 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902015/ [6902017] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} time: 00:26:49 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902017/ [6902019] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:28:01 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902019/ [6902021] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:32:35 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902021/ [6902022] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:59:23 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902022/ [6902023] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} time: 00:44:02 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902023/ [6902024] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:16:54 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902024/ [6902026] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:12:23 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902026/ [6902027] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:12:58 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902027/ [6902028] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:35:53 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902028/ [6902029] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} time: 00:15:29 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902029/ [6902030] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:15:27 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902030/ [6902031] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:25:28 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902031/ [6902033] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment} time: 00:22:56 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902033/ [6902035] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902035/ [6902036] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} time: 00:39:21 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902036/ [6902037] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:49:04 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902037/ [6902038] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 00:38:59 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902038/ [6902039] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair} time: 00:17:38 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902039/ [6902041] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:42 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902041/ [6902042] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:13:14 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902042/ [6902043] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:12:33 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902043/ [6902044] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:27:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902044/ [6902045] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:56 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902045/ [6902046] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} time: 00:12:10 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902046/ [6902047] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:17:26 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902047/ [6902050] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} time: 00:28:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902050/ [6902051] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:10:00 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902051/ [6902052] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} time: 00:22:10 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902052/ [6902053] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:31:23 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902053/ [6902054] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 00:43:30 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902054/ [6902055] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/valgrind} time: 00:51:25 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902055/ [6902056] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 00:58:13 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902056/ [6902060] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} time: 01:51:07 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902060/ [6902061] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:58:51 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902061/ [6902062] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry} time: 00:14:45 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902062/ [6902063] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 00:46:54 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902063/ [6902064] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:23:45 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902064/ [6902065] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:18:10 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902065/ [6902066] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} time: 00:27:24 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902066/ [6902068] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:24:53 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902068/ [6902069] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:15:20 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902069/ [6902071] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:16:25 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902071/ [6902072] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} time: 01:00:45 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902072/ [6902073] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:15:34 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902073/ [6902074] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} time: 00:24:36 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902074/ [6902077] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:13:32 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902077/ [6902078] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python} time: 00:12:54 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902078/ [6902079] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:12:41 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902079/ [6902080] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:55:07 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902080/ [6902081] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} time: 00:13:16 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902081/ [6902082] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:06 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902082/ [6902083] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} time: 00:50:54 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902083/ [6902086] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} time: 00:29:36 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902086/ [6902087] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} time: 00:11:59 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902087/ [6902088] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:41 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902088/ [6902089] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} time: 00:27:54 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902089/ [6902090] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:14:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902090/ [6902092] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota} time: 00:15:39 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902092/ [6902094] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:16:33 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902094/ [6902096] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs} time: 00:11:22 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902096/ [6902098] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} time: 00:13:50 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902098/ [6902099] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:53:07 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902099/ [6902101] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:35:12 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902101/ [6902104] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:47 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902104/ [6902106] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:24:30 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902106/ [6902108] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:11:31 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902108/ [6902109] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:26:05 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902109/ [6902110] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:12:45 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902110/ [6902111] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap} time: 00:13:45 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902111/ [6902112] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} time: 00:23:58 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902112/ [6902113] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:14:14 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902113/ [6902114] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:23 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902114/ [6902115] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:12:27 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902115/ [6902116] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:16:58 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902116/ [6902117] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:12:58 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902117/ [6902118] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} time: 00:22:25 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902118/ [6902120] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:01:56 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902120/ [6902121] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 00:56:36 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902121/ [6902122] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:39:12 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902122/ [6902123] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:19:29 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902123/ [6902124] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:58:19 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902124/ [6902126] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:14 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902126/ [6902127] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 01:36:05 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902127/ [6902128] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:19:41 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902128/ [6902129] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:11:53 info: http://pulpito.ceph.com/teuthology-2022-06-28_03:15:04-fs-quincy-distro-default-smithi/6902129/