2022-04-05T11:11:45.154 INFO:root:teuthology version: 1.1.0-d877b023 2022-04-05T11:11:45.156 INFO:teuthology.results:subset: '8/32' 2022-04-05T11:11:45.156 INFO:teuthology.results:seed: '5566' 2022-04-05T16:43:37.579 INFO:root:teuthology version: 1.1.0-d877b023 2022-04-05T16:43:37.611 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-06T04:44:37.571 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-04-06T04:45:03.800 INFO:scrape:Found 292 jobs 2022-04-06T04:45:21.469 INFO:scrape:Found 15 distinct failure reasons 2022-04-06T04:45:21.471 INFO:scrape:Failure: Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 774719e2-b501-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.8 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' 2022-04-06T04:45:21.471 INFO:scrape:21 jobs: ['6777045', '6776907', '6776801', '6776829', '6776978', '6777033', '6776963', '6776939', '6776884', '6776950', '6776759', '6777004', '6776866', '6776897', '6776814', '6776842', '6776790', '6776856', '6776926', '6777020', '6776991'] 2022-04-06T04:45:21.472 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.473 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/3', 'ranks/5', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/fs/misc}', 'workunit/fs/norstats}', 'workunit/fs/test_o_trunc}', 'workunit/kernel_untar_build}', 'workunit/suites/blogbench}', 'workunit/suites/dbench}', 'workunit/suites/ffsb}', 'workunit/suites/fsstress}', 'workunit/suites/fsx}', 'workunit/suites/fsync-tester}', 'workunit/suites/iogen}', 'workunit/suites/iozone}', 'workunit/suites/pjd}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-04-06T04:45:21.473 INFO:scrape: 2022-04-06T04:45:21.474 INFO:scrape:Failure: "2022-04-05T14:30:09.534994+0000 mon.a (mon.0) 1241 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-04-06T04:45:21.474 INFO:scrape:7 jobs: ['6776933', '6776824', '6776803', '6776855', '6776920', '6776794', '6776962'] 2022-04-06T04:45:21.475 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.475 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{mon-debug', 'overrides/{no_client_pidfile', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/client-limits}', 'tasks/exports}', 'tasks/metrics}', 'tasks/multifs-auth}', 'tasks/strays}', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.476 INFO:scrape: 2022-04-06T04:45:21.476 INFO:scrape:Failure: Command failed on smithi039 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" 2022-04-06T04:45:21.477 INFO:scrape:15 jobs: ['6776766', '6776867', '6776929', '6776810', '6776822', '6776792', '6776837', '6776846', '6776781', '6776894', '6776902', '6776912', '6776960', '6776881', '6776858'] 2022-04-06T04:45:21.477 INFO:scrape:suites intersection: [] 2022-04-06T04:45:21.478 INFO:scrape:suites union: ['1-client', '1-mount', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-upgrade', '2-workload/dbench', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/pacific}}', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-compat_client', '4-verify}', '4-verify}}', '5-client-sanity}}', 'centos_8.stream_container_tools', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'distro/single-container-host}', 'fs/cephadm/multivolume/{0-start', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/no', 'overrides/{multimds/yes', 'overrides/{pg-warn', 'pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.478 INFO:scrape: 2022-04-06T04:45:21.479 INFO:scrape:Failure: Command failed on smithi064 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" 2022-04-06T04:45:21.479 INFO:scrape:1 jobs: ['6776961'] 2022-04-06T04:45:21.480 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.480 INFO:scrape: 2022-04-06T04:45:21.481 INFO:scrape:Dead: b'2022-04-05T19:28:28.439 DEBUG:teuthology.exit:Finished running handlers' 2022-04-06T04:45:21.481 INFO:scrape:ceph version 17.1.0-163-g4e244311 (4e244311a4a30b157b41694d9cd9c9a9ecef285f) quincy (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2c396bfce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x563a613d7c6c] 5: (KernelDevice::_aio_thread()+0x1275) [0x563a6201d195] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x563a6202a6f1] 7: /lib64/libpthread.so.0(+0x81cf) [0x7f2c396b51cf] 8: clone() 2022-04-06T04:45:21.482 INFO:scrape:2 jobs: ['6776871', '6777015'] 2022-04-06T04:45:21.482 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/snaps/{begin/{0-install', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{whitelist_health', 'tasks/workunit/snaps}', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.483 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/snaps/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{whitelist_health', 'tasks/workunit/snaps}', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.483 INFO:scrape: 2022-04-06T04:45:21.484 INFO:scrape:Failure: Command failed (workunit test suites/iozone.sh) on smithi131 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e244311a4a30b157b41694d9cd9c9a9ecef285f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/iozone.sh' 2022-04-06T04:45:21.484 INFO:scrape:2 jobs: ['6776938', '6776761'] 2022-04-06T04:45:21.485 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{centos_8}', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.485 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/iozone}}', '2-workunit/suites/iozone}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.486 INFO:scrape: 2022-04-06T04:45:21.486 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-04-06T04:45:21.486 INFO:scrape:3 jobs: ['6776840', '6777016', '6777023'] 2022-04-06T04:45:21.487 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.487 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.488 INFO:scrape: 2022-04-06T04:45:21.488 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi104 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e244311a4a30b157b41694d9cd9c9a9ecef285f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2022-04-06T04:45:21.489 INFO:scrape:2 jobs: ['6776783', '6776833'] 2022-04-06T04:45:21.489 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{centos_8}', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.490 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'fs/traceless/{begin/{0-install', 'fs/verify/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'overrides/{mon-debug', 'ranks/1', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'tasks/dbench', 'traceless/50pc}', 'validater/lockdep}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.490 INFO:scrape: 2022-04-06T04:45:21.491 INFO:scrape:Failure: SSH connection to smithi154 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2022-04-06T04:45:21.491 INFO:scrape:2 jobs: ['6777021', '6776954'] 2022-04-06T04:45:21.492 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.492 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.493 INFO:scrape: 2022-04-06T04:45:21.493 INFO:scrape:Failure: Command failed on smithi049 with status 1: 'sudo ceph --cluster ceph osd crush tunables default' 2022-04-06T04:45:21.494 INFO:scrape:1 jobs: ['6776924'] 2022-04-06T04:45:21.494 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/verify/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{mon-debug', 'ranks/1', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.495 INFO:scrape: 2022-04-06T04:45:21.495 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e244311a4a30b157b41694d9cd9c9a9ecef285f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2022-04-06T04:45:21.496 INFO:scrape:1 jobs: ['6776948'] 2022-04-06T04:45:21.496 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.497 INFO:scrape: 2022-04-06T04:45:21.497 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/pjd.sh 2022-04-06T04:45:21.498 INFO:scrape:1 jobs: ['6776798'] 2022-04-06T04:45:21.498 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.499 INFO:scrape: 2022-04-06T04:45:21.499 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) 2022-04-06T04:45:21.500 INFO:scrape:1 jobs: ['6776941'] 2022-04-06T04:45:21.500 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/client-readahead}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.500 INFO:scrape: 2022-04-06T04:45:21.501 INFO:scrape:Failure: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) 2022-04-06T04:45:21.501 INFO:scrape:1 jobs: ['6776782'] 2022-04-06T04:45:21.502 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2022-04-06T04:45:21.502 INFO:scrape: 2022-04-06T04:45:21.503 INFO:scrape:Failure: Command failed on smithi025 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e26caf42-b4f3-11ec-8c36-001a4aab830c -- ceph orch daemon add osd smithi025:vg_nvme/lv_2' 2022-04-06T04:45:21.503 INFO:scrape:1 jobs: ['6776949'] 2022-04-06T04:45:21.504 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T04:45:21.504 INFO:scrape: 2022-04-06T04:45:21.519 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/ failed: 57 dead: 2 running: 2 waiting: 0 queued: 0 passed: 231 Fail ================================================================= [6776759] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} ----------------------------------------------------------------- time: 00:10:37 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776759/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776759/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ff98ec937b34197997a9b3a9a56c23a Command failed on smithi002 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid ea003c08-b4d3-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776761] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:11:18 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776761/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776761/ sentry: https://sentry.ceph.com/organizations/ceph/?query=864121ada68247248e60a827b06fe9b6 Command failed (workunit test suites/iozone.sh) on smithi152 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e244311a4a30b157b41694d9cd9c9a9ecef285f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh' [6776766] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:15:44 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776766/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776766/ sentry: https://sentry.ceph.com/organizations/ceph/?query=720e97342db146378334cb2e74040cf1 Command failed on smithi039 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" [6776781] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:08:48 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776781/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776781/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8779b7f57662487a9629f3eea42fced7 Command failed on smithi117 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.1" [6776782] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:15:29 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776782/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776782/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da97c1b961ff481eb11fab4a96926f2b Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) [6776783] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 00:09:13 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776783/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=588ca9cc3afc4a0ca84fc6cf95f0054a Command failed (workunit test suites/dbench.sh) on smithi104 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e244311a4a30b157b41694d9cd9c9a9ecef285f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [6776790] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} ----------------------------------------------------------------- time: 00:10:18 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776790/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776790/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b68bf341d2c64e5a872d7324c873e4f8 Command failed on smithi146 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 3692e15e-b4d6-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.146 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776792] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:12:39 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776792/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776792/ sentry: https://sentry.ceph.com/organizations/ceph/?query=476800bbda2746dca3980851bbeba76c Command failed on smithi139 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.1" [6776794] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} ----------------------------------------------------------------- time: 00:22:07 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776794/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776794/ "2022-04-05T11:52:00.845588+0000 mon.a (mon.0) 383 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6776798] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 06:24:27 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776798/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776798/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23aeca9731f449d395fc6f42995226aa Command failed (workunit test suites/pjd.sh) on smithi080 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e244311a4a30b157b41694d9cd9c9a9ecef285f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6776801] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} ----------------------------------------------------------------- time: 00:10:54 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776801/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776801/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b49aef3d99b0450d98c150506376e674 Command failed on smithi039 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid a1a4675a-b4d7-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.39 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776803] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:04:31 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776803/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776803/ "2022-04-05T12:02:48.140697+0000 mon.a (mon.0) 398 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log [6776810] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:12:41 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776810/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df26e21c95fa4c3798712a3f0a1f38db Command failed on smithi025 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" [6776814] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:28 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776814/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776814/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e380e22089a941cb9485051452da3e97 Command failed on smithi119 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 28a1da0c-b4d9-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.119 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776822] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:14:10 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776822/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776822/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76038802669a4ac1bcb0311452ba239e Command failed on smithi117 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.1" [6776824] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:19:57 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776824/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776824/ "2022-04-05T12:27:44.418274+0000 mon.a (mon.0) 1680 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [6776829] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} ----------------------------------------------------------------- time: 00:13:11 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776829/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776829/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2922dd26564496c99b722348924f8f7 Command failed on smithi016 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 9f653a92-b4db-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776833] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 00:07:45 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776833/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776833/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cf2d1ef6adb74f0f94289d634f06c126 Command failed (workunit test suites/dbench.sh) on smithi106 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e244311a4a30b157b41694d9cd9c9a9ecef285f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [6776837] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:08:42 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776837/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776837/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b097edaa27ec4bed902b91c8176eb269 Command failed on smithi071 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" [6776840] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:17:36 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776840/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776840/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5af29a9e815642c4b8ecd525c6590d55 reached maximum tries (90) after waiting for 540 seconds [6776842] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:43 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776842/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776842/ sentry: https://sentry.ceph.com/organizations/ceph/?query=736f1497a2aa4ed098ef85645d8c9424 Command failed on smithi049 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid d28e625c-b4dd-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.49 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776846] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:12:24 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776846/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776846/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43bfe5859695400ca93a6ca2ce735aaf Command failed on smithi115 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.1" [6776855] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} ----------------------------------------------------------------- time: 00:23:33 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776855/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776855/ "2022-04-05T12:53:33.168749+0000 mon.a (mon.0) 986 : cluster [WRN] Health check failed: Degraded data redundancy: 2/50 objects degraded (4.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6776856] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:10:15 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776856/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776856/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6cbe8a8377a47dbb14576d907b2bf8b Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid baa0f13a-b4df-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.81 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776858] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:08:34 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776858/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776858/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b8ccdad657e415e80bafad121ac0168 Command failed on smithi157 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" [6776866] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:40 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776866/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776866/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e7aaa2916f448cda34830369045ba60 Command failed on smithi049 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 84df376c-b4e1-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.49 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776867] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:14:18 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776867/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776867/ sentry: https://sentry.ceph.com/organizations/ceph/?query=321f0888acee472fa7e490a256bf5ccb Command failed on smithi053 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.1" [6776881] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:13:10 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776881/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776881/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef96877f83e6497b89c161d09b94f25b Command failed on smithi043 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" [6776884] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} ----------------------------------------------------------------- time: 00:10:38 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776884/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776884/ sentry: https://sentry.ceph.com/organizations/ceph/?query=673d246ef33f41faa144d4449a1abd1c Command failed on smithi090 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 5a27092a-b4e4-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.90 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776894] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:11:58 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776894/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55099ecef5be4234b4ea1ac50cc2cb8c Command failed on smithi089 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.1" [6776897] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} ----------------------------------------------------------------- time: 00:11:17 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776897/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776897/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a4493c641524fe7953ccf828010334c Command failed on smithi059 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 3296d960-b4e6-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.59 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776902] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:08:04 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776902/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776902/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffa920fe46314d869f18939d9190c7d4 Command failed on smithi066 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" [6776907] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 00:10:13 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776907/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776907/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b46f6dc869f54727828eb2d21e886708 Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 4eea97f8-b4e8-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.81 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776912] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:14:27 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776912/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776912/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f87022a9a004940a0a4b46b5f07b413 Command failed on smithi152 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" [6776920] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:25:26 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776920/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776920/ "2022-04-05T14:14:37.143586+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [6776924] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:55:03 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776924/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4a444695e774408950be88d021f8884 Command failed on smithi049 with status 1: 'sudo ceph --cluster ceph osd crush tunables default' [6776926] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:21 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776926/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f5581d98481749178822387033329330 Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 50f38eda-b4eb-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.81 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776929] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:08:37 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776929/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776929/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f70b94e361064a71a302c6c6930c4a27 Command failed on smithi153 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" [6776933] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} ----------------------------------------------------------------- time: 00:16:42 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776933/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776933/ "2022-04-05T14:30:09.534994+0000 mon.a (mon.0) 1241 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6776938] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:11:15 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776938/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776938/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2700498c40e84b7ab9f2dfb0f7a9941b Command failed (workunit test suites/iozone.sh) on smithi131 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e244311a4a30b157b41694d9cd9c9a9ecef285f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/iozone.sh' [6776939] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} ----------------------------------------------------------------- time: 00:10:44 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776939/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776939/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c26c2b3ec2b747148973ce805d880c0f Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 506b1e48-b4f4-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.8 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776941] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} ----------------------------------------------------------------- time: 00:10:01 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776941/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3ab0f861bc249c9875b063b9f4128c9 Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) [6776948] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:20:30 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776948/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776948/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57a60f2267dc48bfbc0bee92ddb3c664 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e244311a4a30b157b41694d9cd9c9a9ecef285f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [6776949] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:08:41 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776949/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776949/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56369c06915b4540a34da742665d80c5 Command failed on smithi025 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e26caf42-b4f3-11ec-8c36-001a4aab830c -- ceph orch daemon add osd smithi025:vg_nvme/lv_2' [6776950] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} ----------------------------------------------------------------- time: 00:11:39 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776950/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776950/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eab64e7fdfe14b908ca35f19e2c2305a Command failed on smithi002 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid e0fb5144-b4f4-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776960] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:12:38 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776960/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776960/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b9bb39afaae4483bb3c13e4cc9473c6e Command failed on smithi093 with status 2: "cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\\0' >/home/ubuntu/cephtest/workunits.list.client.0" [6776961] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:08:55 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776961/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776961/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7fad7ff7677e4200adebafa1f72b773d Command failed on smithi064 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6776962] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} ----------------------------------------------------------------- time: 00:45:17 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776962/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776962/ "2022-04-05T16:02:21.551181+0000 mon.a (mon.0) 3829 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [6776963] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} ----------------------------------------------------------------- time: 00:10:48 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776963/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776963/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3d5a36ae2054ed8afb2628dba502969 Command failed on smithi053 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 1f04f750-b4f6-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.53 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776978] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:40 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776978/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776978/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de227260569c4ed8bfe66f6d88d72866 Command failed on smithi033 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid d149368c-b4f7-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.33 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6776991] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:57 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776991/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776991/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5e60ce0257740e8974870c930eef7ff Command failed on smithi053 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 038506a2-b4f9-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.53 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6777004] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:50 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777004/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777004/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7e706640ae24493a170ecb9b5384fdc Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid d8d8a0ce-b4fa-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.6 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6777016] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:18:31 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777016/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777016/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b34ab559883d46db817664ed610208c9 reached maximum tries (90) after waiting for 540 seconds [6777020] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:42 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777020/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777020/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d855ac87437946afa7447d975ba52c9e Command failed on smithi016 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 5e6faae6-b4fd-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6777023] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:15:53 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777023/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777023/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38d373cdc8ef4e72ad120c70d965f167 reached maximum tries (90) after waiting for 540 seconds [6777033] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} ----------------------------------------------------------------- time: 00:11:07 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777033/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777033/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c9e8a7230a94c73acd1839d83c228d7 Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 09ef1c6a-b500-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6777045] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:11:05 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777045/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777045/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9ab84bec0dd4e4e9a27c11459da0bbe Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:4e244311a4a30b157b41694d9cd9c9a9ecef285f -v bootstrap --fsid 774719e2-b501-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.8 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' Dead ================================================================= [6776954] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:28:21 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776954/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776954/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53cc73ad03a64231b55e43894ecb0834 SSH connection to smithi017 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6777021] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:25:31 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777021/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777021/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3d24daaab3945c88bed91ae2d78df37 SSH connection to smithi154 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' Running ================================================================= [6776871] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776871/ [6777015] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777015/ Pass ================================================================= [6776760] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} time: 00:10:23 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776760/ [6776762] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:22:25 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776762/ [6776763] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:11:01 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776763/ [6776764] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:14:09 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776764/ [6776765] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i24137} time: 00:08:27 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776765/ [6776767] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full} time: 00:13:50 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776767/ [6776768] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} time: 00:16:20 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776768/ [6776769] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client} time: 00:07:58 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776769/ [6776770] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 01:53:44 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776770/ [6776771] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} time: 00:21:50 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776771/ [6776772] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:29:12 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776772/ [6776773] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:17:33 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776773/ [6776774] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:55:34 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776774/ [6776775] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:09:56 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776775/ [6776776] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell} time: 00:31:35 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776776/ [6776777] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:41:16 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776777/ [6776778] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:03:24 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776778/ [6776779] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} time: 00:06:55 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776779/ [6776780] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:19:49 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776780/ [6776784] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 00:50:24 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776784/ [6776785] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:23:31 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776785/ [6776786] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:56:33 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776786/ [6776787] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} time: 00:48:16 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776787/ [6776788] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:01:33 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776788/ [6776789] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry} time: 00:13:10 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776789/ [6776791] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:18:05 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776791/ [6776793] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:08:45 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776793/ [6776795] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:30:40 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776795/ [6776796] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:12:19 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776796/ [6776797] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} time: 01:04:21 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776797/ [6776799] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} time: 00:16:55 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776799/ [6776800] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:36:05 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776800/ [6776802] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:07:38 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776802/ [6776804] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} time: 00:13:49 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776804/ [6776805] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} time: 00:22:53 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776805/ [6776806] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:14:39 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776806/ [6776807] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:08 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776807/ [6776808] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} time: 00:24:26 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776808/ [6776809] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} time: 00:10:35 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776809/ [6776811] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:08:57 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776811/ [6776812] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:10:34 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776812/ [6776813] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:10:53 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776813/ [6776815] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:13:13 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776815/ [6776816] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota} time: 00:08:14 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776816/ [6776817] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:23:38 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776817/ [6776818] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:15:34 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776818/ [6776819] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:45:03 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776819/ [6776820] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs} time: 00:10:01 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776820/ [6776821] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:31:07 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776821/ [6776823] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:09:05 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776823/ [6776825] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:46:42 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776825/ [6776826] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:03:02 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776826/ [6776827] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind} time: 01:42:00 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776827/ [6776828] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} time: 00:37:56 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776828/ [6776830] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:15:44 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776830/ [6776831] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} time: 00:19:16 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776831/ [6776832] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:02:20 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776832/ [6776834] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:09:32 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776834/ [6776835] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:22:15 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776835/ [6776836] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:16:48 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776836/ [6776838] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap} time: 00:13:57 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776838/ [6776839] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} time: 00:22:14 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776839/ [6776841] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:16:41 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776841/ [6776843] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} time: 00:45:17 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776843/ [6776844] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:53:23 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776844/ [6776845] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 01:14:46 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776845/ [6776847] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:55:43 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776847/ [6776848] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} time: 00:23:42 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776848/ [6776849] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:12:20 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776849/ [6776850] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/ino_release_cb} time: 00:08:11 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776850/ [6776851] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:25 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776851/ [6776852] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 00:41:32 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776852/ [6776853] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:23:19 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776853/ [6776854] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:11:04 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776854/ [6776857] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:13:12 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776857/ [6776859] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:03:46 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776859/ [6776860] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:14:48 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776860/ [6776861] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration} time: 00:15:19 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776861/ [6776862] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:25:20 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776862/ [6776863] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} time: 01:06:52 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776863/ [6776864] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:50:59 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776864/ [6776865] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay} time: 00:10:02 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776865/ [6776868] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:14:38 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776868/ [6776869] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} time: 00:10:22 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776869/ [6776870] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:44:28 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776870/ [6776872] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:09:40 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776872/ [6776873] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/lockdep} time: 01:13:04 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776873/ [6776874] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:17:51 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776874/ [6776875] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} time: 00:24:05 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776875/ [6776876] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/dir-max-entries} time: 00:12:11 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776876/ [6776877] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:11:16 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776877/ [6776878] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:28:05 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776878/ [6776879] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:12:47 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776879/ [6776880] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota} time: 00:12:26 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776880/ [6776882] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:32:11 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776882/ [6776883] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:53:52 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776883/ [6776885] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:14:49 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776885/ [6776886] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:09:54 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776886/ [6776887] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:09:59 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776887/ [6776888] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:55:25 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776888/ [6776889] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} time: 00:12:50 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776889/ [6776890] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} time: 00:30:23 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776890/ [6776891] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:15:02 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776891/ [6776892] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 01:08:08 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776892/ [6776893] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:14:40 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776893/ [6776895] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} time: 00:47:24 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776895/ [6776896] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:09:21 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776896/ [6776898] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:13:52 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776898/ [6776899] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} time: 00:10:34 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776899/ [6776900] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:21:28 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776900/ [6776901] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:16:56 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776901/ [6776903] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 03:02:52 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776903/ [6776904] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} time: 00:21:57 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776904/ [6776905] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} time: 00:20:33 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776905/ [6776906] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:18:37 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776906/ [6776908] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:10:34 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776908/ [6776909] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} time: 00:08:38 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776909/ [6776910] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 02:05:10 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776910/ [6776911] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:36:59 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776911/ [6776913] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:08:30 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776913/ [6776914] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} time: 00:41:19 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776914/ [6776915] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} time: 00:09:59 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776915/ [6776916] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:09:26 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776916/ [6776917] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/whitelist_health} time: 00:14:39 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776917/ [6776918] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}} time: 00:10:34 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776918/ [6776919] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:18:09 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776919/ [6776921] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:09:23 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776921/ [6776922] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:57:49 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776922/ [6776923] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:14:46 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776923/ [6776925] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:17:15 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776925/ [6776927] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:13:58 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776927/ [6776928] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush} time: 00:09:49 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776928/ [6776930] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:46:57 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776930/ [6776931] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} time: 00:55:22 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776931/ [6776932] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:26:25 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776932/ [6776934] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} time: 00:24:00 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776934/ [6776935] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:16:24 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776935/ [6776936] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:25:56 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776936/ [6776937] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:13:07 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776937/ [6776940] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:07:22 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776940/ [6776942] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:08:57 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776942/ [6776943] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} time: 00:36:28 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776943/ [6776944] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:14:04 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776944/ [6776945] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} time: 00:33:05 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776945/ [6776946] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:21:55 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776946/ [6776947] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:15:14 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776947/ [6776951] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} time: 00:16:46 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776951/ [6776952] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:51:00 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776952/ [6776953] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 01:01:37 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776953/ [6776955] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:13:00 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776955/ [6776956] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:09:22 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776956/ [6776957] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} time: 00:24:54 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776957/ [6776958] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:08:34 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776958/ [6776959] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:23:31 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776959/ [6776964] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:11:58 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776964/ [6776965] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:08:01 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776965/ [6776966] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:51:36 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776966/ [6776967] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:29:51 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776967/ [6776968] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 00:56:27 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776968/ [6776969] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/lockdep} time: 00:45:01 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776969/ [6776970] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 00:47:51 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776970/ [6776971] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} time: 00:24:15 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776971/ [6776972] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} time: 00:12:30 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776972/ [6776973] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:38:06 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776973/ [6776974] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:40 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776974/ [6776975] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 00:43:18 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776975/ [6776976] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:54:08 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776976/ [6776977] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment} time: 00:16:23 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776977/ [6776979] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:20:45 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776979/ [6776980] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:09:33 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776980/ [6776981] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} time: 00:30:12 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776981/ [6776982] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair} time: 00:19:03 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776982/ [6776983] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:21:03 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776983/ [6776984] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:13:40 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776984/ [6776985] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:39:34 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776985/ [6776986] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:20:18 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776986/ [6776987] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:26:03 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776987/ [6776988] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:07 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776988/ [6776989] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} time: 00:10:13 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776989/ [6776990] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:56:07 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776990/ [6776992] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:11:52 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776992/ [6776993] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python} time: 00:09:00 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776993/ [6776994] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:11:48 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776994/ [6776995] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:38:12 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776995/ [6776996] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:49 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776996/ [6776997] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} time: 00:20:01 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776997/ [6776998] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:24:11 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776998/ [6776999] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:18:52 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6776999/ [6777000] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:59 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777000/ [6777001] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} time: 00:23:20 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777001/ [6777002] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry} time: 00:10:12 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777002/ [6777003] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:14:05 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777003/ [6777005] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:14:22 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777005/ [6777006] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:36 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777006/ [6777007] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} time: 00:20:29 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777007/ [6777008] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:22:08 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777008/ [6777009] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:14:27 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777009/ [6777010] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:36:56 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777010/ [6777011] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:31:38 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777011/ [6777012] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} time: 00:16:09 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777012/ [6777013] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} time: 00:18:35 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777013/ [6777014] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} time: 00:25:24 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777014/ [6777017] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/valgrind} time: 02:00:53 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777017/ [6777018] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} time: 00:39:18 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777018/ [6777019] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:00 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777019/ [6777022] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} time: 00:13:57 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777022/ [6777024] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:24:42 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777024/ [6777025] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:16:21 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777025/ [6777026] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:22:19 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777026/ [6777027] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} time: 00:08:41 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777027/ [6777028] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:12:40 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777028/ [6777029] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:39 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777029/ [6777030] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:16:24 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777030/ [6777031] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:16:18 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777031/ [6777032] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:10 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777032/ [6777034] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota} time: 00:10:53 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777034/ [6777035] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:49:42 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777035/ [6777036] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 01:11:16 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777036/ [6777037] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:46:08 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777037/ [6777038] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:45:14 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777038/ [6777039] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:02:31 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777039/ [6777040] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs} time: 00:12:30 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777040/ [6777041] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:20 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777041/ [6777042] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 00:44:13 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777042/ [6777043] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:20:09 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777043/ [6777044] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:08:57 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777044/ [6777046] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} time: 00:16:46 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777046/ [6777047] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:12:24 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777047/ [6777048] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 00:57:29 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777048/ [6777049] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:15:52 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777049/ [6777050] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:19:25 info: http://pulpito.ceph.com/teuthology-2022-04-05_03:15:03-fs-quincy-distro-default-smithi/6777050/