2022-01-11T12:30:33.698 INFO:root:teuthology version: 1.1.0-00e09f05 2022-01-11T12:30:33.700 INFO:teuthology.results:subset: '16/32' 2022-01-11T12:30:33.700 INFO:teuthology.results:seed: '6499' 2022-01-11T18:13:36.080 INFO:root:teuthology version: 1.1.0-00e09f05 2022-01-11T18:13:36.108 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-01-12T04:41:33.909 INFO:teuthology.results:Tests finished! gathering results... 2022-01-12T04:42:06.425 INFO:scrape:Found 327 jobs 2022-01-12T04:42:16.531 INFO:scrape:Found 11 distinct failure reasons 2022-01-12T04:42:16.532 INFO:scrape:Failure: Command failed on smithi018 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' 2022-01-12T04:42:16.532 INFO:scrape:68 jobs 2022-01-12T04:42:16.534 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}'] 2022-01-12T04:42:16.534 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', 'clusters/1-mds-2-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8.stream}', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/functional/{begin', 'fs/mixed-clients/{begin', 'fs/multiclient/{begin', 'fs/multifs/{begin', 'fs/snaps/{begin', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'fs/verify/{begin', 'fs/volumes/{begin', 'fs/workload/{begin', 'k-testing}', 'kclient-overrides/{distro/testing/{flavor/ubuntu_latest', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'overrides/{mon-debug', 'overrides/{no_client_pidfile', 'overrides/{osd-asserts', 'overrides/{whitelist_health', 'overrides}}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/admin}', 'tasks/asok_dump_tree}', 'tasks/backtrace}', 'tasks/client-limits}', 'tasks/data-scan}', 'tasks/failover}', 'tasks/fsstress', 'tasks/ior-shared-file}', 'tasks/journal-repair}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/metrics}', 'tasks/multifs-auth}', 'tasks/quota}', 'tasks/snap-schedule}', 'tasks/truncate_delay}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/clone}}', 'thrashosds-health', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/fs/misc}', 'workunit/fs/norstats}', 'workunit/fs/test_o_trunc}', 'workunit/kernel_untar_build}', 'workunit/suites/blogbench}', 'workunit/suites/dbench}', 'workunit/suites/ffsb}', 'workunit/suites/fsstress}', 'workunit/suites/fsx}', 'workunit/suites/fsync-tester}', 'workunit/suites/iogen}', 'workunit/suites/iozone}', 'workunit/suites/pjd}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-01-12T04:42:16.535 INFO:scrape: 2022-01-12T04:42:16.535 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-01-12T04:42:16.536 INFO:scrape:3 jobs: ['6606959', '6606983', '6606776'] 2022-01-12T04:42:16.536 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-12T04:42:16.536 INFO:scrape:suites union: ['2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8.stream}', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-12T04:42:16.537 INFO:scrape: 2022-01-12T04:42:16.537 INFO:scrape:Failure: {'Failure object was': {'smithi046.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} 2022-01-12T04:42:16.538 INFO:scrape:2 jobs: ['6606690', '6606716'] 2022-01-12T04:42:16.538 INFO:scrape:suites intersection: ['conf/{client', 'distro/{ubuntu_latest}', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-12T04:42:16.539 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_snaptests}}', '2-workunit/suites/fsstress}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-12T04:42:16.539 INFO:scrape: 2022-01-12T04:42:16.540 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi002 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fa08f32a5319df9b2ed0de99714cc34db0639162 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2022-01-12T04:42:16.540 INFO:scrape:1 jobs: ['6606710'] 2022-01-12T04:42:16.540 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}', 'wsync/{no}}'] 2022-01-12T04:42:16.541 INFO:scrape: 2022-01-12T04:42:16.541 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2022-01-12T04:42:16.542 INFO:scrape:1 jobs: ['6606694'] 2022-01-12T04:42:16.542 INFO:scrape:suites: ['centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2022-01-12T04:42:16.542 INFO:scrape: 2022-01-12T04:42:16.543 INFO:scrape:Failure: Command failed on smithi067 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:59482e5f2889b3d141965aac01ab518a594b9799 pull' 2022-01-12T04:42:16.543 INFO:scrape:1 jobs: ['6606678'] 2022-01-12T04:42:16.544 INFO:scrape:suites: ['1-mount', '2-workload/dbench', 'distro/centos_latest}', 'fs/cephadm/multivolume/{0-start'] 2022-01-12T04:42:16.544 INFO:scrape: 2022-01-12T04:42:16.545 INFO:scrape:Dead: b'2022-01-12T03:53:53.042 DEBUG:teuthology.exit:Finished running handlers' 2022-01-12T04:42:16.545 INFO:scrape:3 jobs: ['6606911', '6606757', '6606946'] 2022-01-12T04:42:16.545 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-12T04:42:16.546 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-upgrade-with-workload', '3-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8.stream}', 'distro/{centos_8}', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{begin', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{pg-warn', 'ranks/3', 'ranks/5', 'rhel_8}', 'roles', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'syntax', 'tasks/{0-check-counter', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/kernel_untar_build}', 'workunit/suites/blogbench}', 'wsync/{no}}'] 2022-01-12T04:42:16.546 INFO:scrape: 2022-01-12T04:42:16.547 INFO:scrape:Failure: Command failed on smithi002 with status 1: 'sudo cp /boot/grub/grub.conf /tmp/tmp.J3B8c77NGH' 2022-01-12T04:42:16.547 INFO:scrape:1 jobs: ['6606905'] 2022-01-12T04:42:16.548 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/verify/{begin', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{mon-debug', 'ranks/3', 'session_timeout', 'tasks/dbench', 'validater/lockdep}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-12T04:42:16.548 INFO:scrape: 2022-01-12T04:42:16.548 INFO:scrape:Failure: 2022-01-12T04:42:16.549 INFO:scrape:1 jobs: ['6606904'] 2022-01-12T04:42:16.549 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-upgrade-with-workload', '3-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-12T04:42:16.550 INFO:scrape: 2022-01-12T04:42:16.550 INFO:scrape:Failure: SSH connection to smithi159 was lost: 'sudo yum upgrade -y linux-firmware' 2022-01-12T04:42:16.551 INFO:scrape:1 jobs: ['6606903'] 2022-01-12T04:42:16.551 INFO:scrape:suites: ['2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin', 'k-testing}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-12T04:42:16.551 INFO:scrape: 2022-01-12T04:42:16.552 INFO:scrape:Failure: "2022-01-11T18:36:06.555954+0000 mon.a (mon.0) 298 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-01-12T04:42:16.552 INFO:scrape:1 jobs: ['6607002'] 2022-01-12T04:42:16.553 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/pjd}', 'wsync/{no}}'] 2022-01-12T04:42:16.553 INFO:scrape: 2022-01-12T04:42:16.566 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/ failed: 77 dead: 6 running: 0 waiting: 0 queued: 0 passed: 244 Fail ================================================================= [6606678] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest} ----------------------------------------------------------------- time: 00:09:29 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606678/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606678/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d1433ffed074280b9cba297ae1c8f8d Command failed on smithi067 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:59482e5f2889b3d141965aac01ab518a594b9799 pull' [6606694] fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:19:46 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606694/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606694/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29ae1a29366d43adb1089aa3129c098f Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [6606704] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606704/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=075a6262a22643518db585cd0375ec56 Command failed on smithi152 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606705] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606705/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606705/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9dd8307aa4bf4b04b6deda87cd474af5 Command failed on smithi102 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606710] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} ----------------------------------------------------------------- time: 01:04:34 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606710/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606710/ sentry: https://sentry.ceph.com/organizations/ceph/?query=badbd723ffc4429a86b0d65da1f14371 Command failed (workunit test suites/ffsb.sh) on smithi002 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fa08f32a5319df9b2ed0de99714cc34db0639162 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [6606713] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606713/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606713/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9eb538a1bb24a28a7367023d204a0ed Command failed on smithi067 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606714] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606714/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606714/ sentry: https://sentry.ceph.com/organizations/ceph/?query=86955a70afaf4543a4c335f0484d2e9d Command failed on smithi142 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606717] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606717/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606717/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b0efca9f7e84fa8a42e1d6b08eb9e97 Command failed on smithi096 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606722] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606722/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606722/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e763946c4d54cf281a1c7be5173de72 Command failed on smithi059 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606731] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606731/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606731/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d672d7b7713946d0b59cad26a3082504 Command failed on smithi057 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606736] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606736/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606736/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a66a94da323f4af5b7b40137754bdb63 Command failed on smithi064 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606740] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606740/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606740/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef3171bea9d0493b8569279f4b9fc510 Command failed on smithi120 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606747] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606747/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606747/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4066efee11c4a7a9c81327f272c3b58 Command failed on smithi012 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606748] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606748/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54144f6f240c4c629b3eaa8387d2014c Command failed on smithi033 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606749] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606749/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606749/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55e2b28873384a2da599d7bd1438eda0 Command failed on smithi064 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606751] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606751/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606751/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15df42a1bd0a4be49c7c0d66acc9e279 Command failed on smithi170 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606760] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606760/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606760/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0310f6476405426b8185ddf2c6a23d03 Command failed on smithi018 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606763] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606763/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606763/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3cdd4a0f71f0471bae583ed4265c833f Command failed on smithi067 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606773] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606773/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606773/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8aa2d7574cda4acaaf24894d58002efd Command failed on smithi118 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606774] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606774/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606774/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49e7486b3af74d22911ebb7f26ce1f8d Command failed on smithi027 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606776] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:18:11 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606776/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606776/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0df0ac951cf940be92403cf066bced6b reached maximum tries (90) after waiting for 540 seconds [6606779] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606779/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=235dd1dba4c1422889dca84af2022d14 Command failed on smithi135 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606783] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606783/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=faf92d4c167540c59d4cad28bbbc3609 Command failed on smithi067 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606790] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606790/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606790/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0fb3020a5f4a454f90228673e69cccb7 Command failed on smithi029 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606791] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606791/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62948af036084ac5bfe5c61becb88e69 Command failed on smithi037 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606793] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606793/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f1c4ec3805e48cb8168c740e7c70d3d Command failed on smithi161 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606796] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606796/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606796/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f84e64a110b64f1f9f62d464bd402ab7 Command failed on smithi016 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606803] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606803/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606803/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af6074f0cb3b450589eb59dba74e2e5d Command failed on smithi163 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606809] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606809/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606809/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5dc6879dedfa4836829b6480b964de87 Command failed on smithi067 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606810] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606810/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43592d87ca6a44ea99c4e73faf8a4006 Command failed on smithi092 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606821] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606821/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606821/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f54ad962b24a4903a57fedf6fef1bbdc Command failed on smithi067 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606824] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606824/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e09f8c1a3e7a45feb5e194e991dcd720 Command failed on smithi106 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606829] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606829/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606829/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ced33ee3e43484d9a0e41376eb0d798 Command failed on smithi003 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606832] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606832/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606832/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49f1625bc4b548278233a831d5791d98 Command failed on smithi046 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606836] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606836/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606836/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0fb9d4a32d442bab52ec4826c913f33 Command failed on smithi067 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606841] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606841/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d0c35ffeedf4474b5acfb09830c8578 Command failed on smithi002 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606850] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606850/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606850/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7013daae2f7344cbb3827d60db5c1ae6 Command failed on smithi072 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606852] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606852/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606852/ sentry: https://sentry.ceph.com/organizations/ceph/?query=80cbbcaf15e94f2c89193180f060cf2f Command failed on smithi002 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606855] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606855/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606855/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a52bd5671e0468d9d036a706f78421f Command failed on smithi092 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606856] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606856/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606856/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd8865552dac41ff8330027c81790ef9 Command failed on smithi087 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606864] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606864/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab9087e8ba704795957ec4062fa3c587 Command failed on smithi077 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606868] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606868/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e2004287af240eeaa16791641e5e564 Command failed on smithi099 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606871] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606871/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606871/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0067986db6d94714826800226c30d9e5 Command failed on smithi012 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606879] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606879/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606879/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f1153b84fe94356814bef196bbbf69e Command failed on smithi162 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606884] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606884/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606884/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff608f6d963442e2bb32041615ec5aff Command failed on smithi036 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606888] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606888/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606888/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a686635d77664a4ab63a7d0305724f13 Command failed on smithi065 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606893] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606893/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606893/ sentry: https://sentry.ceph.com/organizations/ceph/?query=295ae21b486d4233874ae47fd5251002 Command failed on smithi003 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606897] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606897/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606897/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1fb46205bb394066b8ca18e7f39dc56d Command failed on smithi042 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606898] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606898/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606898/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9278fe80ac5940e7894ddd1db6efd0e4 Command failed on smithi029 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606901] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606901/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606901/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f64e5193b21d411595853b635db27c1e Command failed on smithi132 with status 1: 'sudo yum install -y kernel' [6606902] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606902/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606902/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b27b583b162142b7a501d5cefcda3235 Command failed on smithi022 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606904] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606904/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606904/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ed66885751142d18db0ed97256dcf15 [6606905] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606905/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606905/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3d77f4209e34f389359fa4621f7125f Command failed on smithi002 with status 1: 'sudo cp /boot/grub/grub.conf /tmp/tmp.J3B8c77NGH' [6606906] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606906/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606906/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f264fa639dad4deb87f2adb547854341 Command failed on smithi174 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606907] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606907/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606907/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b653cc873e0b40e48802bde307df49bc Command failed on smithi116 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606916] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606916/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606916/ sentry: https://sentry.ceph.com/organizations/ceph/?query=05009a9799af41cca4ef83303d4ef819 Command failed on smithi061 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606917] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606917/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606917/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de01814d36b940c385c59636d74bfb23 Command failed on smithi038 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606924] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606924/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5021179e5dce4f6d8cce57117b096ff4 Command failed on smithi067 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606928] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606928/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606928/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8250c69971a4eb29b6a75abc20f6442 Command failed on smithi102 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606936] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606936/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606936/ sentry: https://sentry.ceph.com/organizations/ceph/?query=372a507a71084df0b3cdd3f77598f6b9 Command failed on smithi053 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606940] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606940/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606940/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be76f290fe47490bbe1e8c38ca0dff73 Command failed on smithi139 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606942] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606942/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606942/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fab9ea0e5a94af5aa8a50b6763b42cd Command failed on smithi096 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606948] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606948/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606948/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f6720d8e1ba4b2dad64c35c29395de2 Command failed on smithi110 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606950] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606950/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606950/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7dff1d0173294b46ade5e61814aec86b Command failed on smithi002 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606952] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606952/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606952/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38f0dbcf1d8e46b6ae99ccd96b98a3a4 Command failed on smithi079 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606954] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606954/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606954/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8bb4f1d3022498baa1c247e90690141 Command failed on smithi111 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606959] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:23:04 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606959/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606959/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ddf3339b1c7b4fa9a6c1be4d8ac72260 reached maximum tries (90) after waiting for 540 seconds [6606963] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606963/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606963/ sentry: https://sentry.ceph.com/organizations/ceph/?query=273ee6c57db54d4f84922e5404dbe02f Command failed on smithi036 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606964] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606964/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606964/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c95d3306b7544a3f9173102b45535879 Command failed on smithi008 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606977] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606977/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606977/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d516b246d9f437ebd4edaae97c6b88e Command failed on smithi087 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606978] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606978/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606978/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d52635d71324457d8461eddc0c405c47 Command failed on smithi093 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606983] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:22:19 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606983/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606983/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e27c7bb42fe447df9ee1a733814cb98d reached maximum tries (90) after waiting for 540 seconds [6606985] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606985/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606985/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b8b7cc3256540f392db7d3c280456f6 Command failed on smithi039 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606988] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606988/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606988/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2cf399070c264804977c903ec08a07ed Command failed on smithi096 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606994] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606994/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606994/ sentry: https://sentry.ceph.com/organizations/ceph/?query=edfa76391632496f9bdcee5aeaf3d3e0 Command failed on smithi003 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6606998] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606998/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606998/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ca666bda9724314a1ac81a5c32dffa9 Command failed on smithi005 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6607002] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} ----------------------------------------------------------------- time: 00:17:36 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6607002/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6607002/ "2022-01-11T18:36:06.555954+0000 mon.a (mon.0) 298 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log Dead ================================================================= [6606690] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:04:01 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606690/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606690/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2838a2fccafb44ce812ddb5cbe88b932 {'Failure object was': {'smithi046.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2 ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2 ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} [6606716] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:04:03 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606716/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606716/ sentry: https://sentry.ceph.com/organizations/ceph/?query=409b724eb873424da7aa5a35b0ffce91 {'Failure object was': {'smithi058.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2 ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2 ab7c7f497f5e446613531743ae2/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} [6606757] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606757/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606757/ hit max job timeout [6606903] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606903/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606903/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c02809829044f5792d513cba4bfd672 SSH connection to smithi159 was lost: 'sudo yum upgrade -y linux-firmware' [6606911] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606911/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606911/ hit max job timeout [6606946] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606946/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606946/ hit max job timeout Pass ================================================================= [6606676] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:44 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606676/ [6606677] fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i24137} time: 00:11:40 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606677/ [6606679] fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full} time: 00:14:50 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606679/ [6606680] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} time: 00:12:19 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606680/ [6606681] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client} time: 00:10:14 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606681/ [6606682] fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror} time: 01:54:52 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606682/ [6606683] fs/mirror-ha/{begin cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit} time: 00:22:42 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606683/ [6606684] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:43:49 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606684/ [6606685] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:19:55 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606685/ [6606686] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:52:41 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606686/ [6606687] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:13:31 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606687/ [6606688] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell} time: 00:34:02 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606688/ [6606689] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:39:41 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606689/ [6606691] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{centos_8} tasks/fstop} time: 00:10:19 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606691/ [6606692] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:21:40 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606692/ [6606693] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:46 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606693/ [6606695] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep} time: 00:42:47 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606695/ [6606696] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 00:52:47 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606696/ [6606697] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:13:52 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606697/ [6606698] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:40:25 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606698/ [6606699] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 00:36:36 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606699/ [6606700] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:52:08 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606700/ [6606701] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} time: 00:50:41 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606701/ [6606702] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} time: 00:32:11 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606702/ [6606703] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:27:44 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606703/ [6606706] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:14:03 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606706/ [6606707] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} time: 00:13:51 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606707/ [6606708] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} time: 00:53:27 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606708/ [6606709] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:58 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606709/ [6606711] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:14:46 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606711/ [6606712] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:31 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606712/ [6606715] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:16:09 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606715/ [6606718] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:22 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606718/ [6606719] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:19:09 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606719/ [6606720] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} time: 00:14:47 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606720/ [6606721] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:23:02 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606721/ [6606723] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:16:33 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606723/ [6606724] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:41 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606724/ [6606725] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} time: 00:14:20 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606725/ [6606726] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:23:45 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606726/ [6606727] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:21:06 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606727/ [6606728] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:11:58 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606728/ [6606729] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:13 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606729/ [6606730] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:51:42 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606730/ [6606732] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:10:23 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606732/ [6606733] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:26:24 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606733/ [6606734] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush} time: 00:13:54 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606734/ [6606735] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:11:58 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606735/ [6606737] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:17:01 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606737/ [6606738] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} time: 00:16:43 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606738/ [6606739] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:17:23 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606739/ [6606741] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:56:20 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606741/ [6606742] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:10:10 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606742/ [6606743] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} time: 00:24:29 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606743/ [6606744] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind} time: 01:35:44 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606744/ [6606745] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} time: 00:15:48 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606745/ [6606746] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:34:20 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606746/ [6606750] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:50:12 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606750/ [6606752] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:21:18 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606752/ [6606753] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} time: 00:14:26 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606753/ [6606754] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} time: 01:18:20 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606754/ [6606755] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:01 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606755/ [6606756] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:29 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606756/ [6606758] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:16:20 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606758/ [6606759] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} time: 00:32:47 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606759/ [6606761] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:06:51 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606761/ [6606762] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:24 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606762/ [6606764] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:55:07 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606764/ [6606765] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} time: 00:22:40 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606765/ [6606766] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} time: 00:13:41 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606766/ [6606767] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:58:11 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606767/ [6606768] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:23:18 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606768/ [6606769] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:37 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606769/ [6606770] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/ino_release_cb} time: 00:11:03 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606770/ [6606771] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:20 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606771/ [6606772] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} time: 00:18:37 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606772/ [6606775] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:21:29 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606775/ [6606777] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:05 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606777/ [6606778] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:16:50 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606778/ [6606780] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:06:09 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606780/ [6606781] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} time: 00:41:53 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606781/ [6606782] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:25:30 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606782/ [6606784] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 01:01:38 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606784/ [6606785] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:20:56 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606785/ [6606786] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} time: 00:17:03 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606786/ [6606787] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:35 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606787/ [6606788] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:15:44 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606788/ [6606789] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:17:53 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606789/ [6606792] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment} time: 00:20:01 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606792/ [6606794] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} time: 00:11:48 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606794/ [6606795] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:50:08 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606795/ [6606797] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:33 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606797/ [6606798] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/lockdep} time: 01:04:01 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606798/ [6606799] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:16:02 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606799/ [6606800] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:19:18 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606800/ [6606801] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 00:29:51 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606801/ [6606802] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:24:04 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606802/ [6606804] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 00:39:55 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606804/ [6606805] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:35:33 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606805/ [6606806] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:50:52 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606806/ [6606807] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:54:29 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606807/ [6606808] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:15:37 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606808/ [6606811] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:12:08 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606811/ [6606812] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:11:51 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606812/ [6606813] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:15:42 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606813/ [6606814] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} time: 00:09:50 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606814/ [6606815] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} time: 00:58:24 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606815/ [6606816] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:16:47 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606816/ [6606817] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 03:10:12 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606817/ [6606818] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:44:43 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606818/ [6606819] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} time: 00:23:58 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606819/ [6606820] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:22:57 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606820/ [6606822] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} time: 00:16:13 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606822/ [6606823] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:20:35 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606823/ [6606825] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:17 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606825/ [6606826] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry} time: 00:15:49 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606826/ [6606827] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:18:16 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606827/ [6606828] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} time: 00:24:28 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606828/ [6606830] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:16:29 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606830/ [6606831] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 01:02:45 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606831/ [6606833] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} time: 00:24:09 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606833/ [6606834] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:42:32 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606834/ [6606835] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 02:22:23 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606835/ [6606837] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:31 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606837/ [6606838] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:32:59 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606838/ [6606839] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} time: 00:15:54 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606839/ [6606840] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:17:24 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606840/ [6606842] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:15:06 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606842/ [6606843] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} time: 00:16:45 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606843/ [6606844] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} time: 00:15:23 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606844/ [6606845] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:17:41 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606845/ [6606846] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:11:25 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606846/ [6606847] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}} time: 00:11:37 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606847/ [6606848] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:55:55 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606848/ [6606849] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:19:46 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606849/ [6606851] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:37 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606851/ [6606853] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:15:22 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606853/ [6606854] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:45:07 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606854/ [6606857] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} time: 00:14:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606857/ [6606858] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:23:36 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606858/ [6606859] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} time: 01:59:54 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606859/ [6606860] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:14:56 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606860/ [6606861] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:57 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606861/ [6606862] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:24:37 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606862/ [6606863] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:17 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606863/ [6606865] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:54:30 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606865/ [6606866] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:16:07 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606866/ [6606867] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:23:15 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606867/ [6606869] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} time: 01:03:25 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606869/ [6606870] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:17:33 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606870/ [6606872] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:28:10 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606872/ [6606873] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs} time: 00:14:40 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606873/ [6606874] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:14:11 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606874/ [6606875] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:42:45 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606875/ [6606876] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:24:40 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606876/ [6606877] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} time: 00:18:38 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606877/ [6606878] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} time: 00:22:19 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606878/ [6606880] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:26:10 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606880/ [6606881] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:14:46 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606881/ [6606882] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:55 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606882/ [6606883] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:16:58 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606883/ [6606885] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap} time: 00:16:08 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606885/ [6606886] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:15:01 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606886/ [6606887] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:21:04 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606887/ [6606889] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:32:13 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606889/ [6606890] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:17:06 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606890/ [6606891] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 00:28:53 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606891/ [6606892] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:21:28 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606892/ [6606894] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:14:46 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606894/ [6606895] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} time: 00:17:04 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606895/ [6606896] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:20:27 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606896/ [6606899] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} time: 00:25:18 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606899/ [6606900] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:44:13 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606900/ [6606908] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 00:52:24 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606908/ [6606909] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} time: 00:30:20 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606909/ [6606910] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:45:52 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606910/ [6606912] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:30:14 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606912/ [6606913] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:14:39 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606913/ [6606914] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:31:34 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606914/ [6606915] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration} time: 00:18:36 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606915/ [6606918] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:16:59 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606918/ [6606919] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 00:50:49 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606919/ [6606920] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:55:50 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606920/ [6606921] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:35:53 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606921/ [6606922] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:22:37 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606922/ [6606923] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} time: 00:49:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606923/ [6606925] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:26 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606925/ [6606926] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:09:13 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606926/ [6606927] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:16:32 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606927/ [6606929] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:34 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606929/ [6606930] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python} time: 00:15:08 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606930/ [6606931] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:12:10 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606931/ [6606932] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} time: 00:18:41 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606932/ [6606933] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:10:38 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606933/ [6606934] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/dir-max-entries} time: 00:16:29 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606934/ [6606935] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:17:24 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606935/ [6606937] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:16:24 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606937/ [6606938] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:15:09 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606938/ [6606939] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota} time: 00:22:24 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606939/ [6606941] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:25:08 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606941/ [6606943] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:41:09 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606943/ [6606944] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 00:46:19 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606944/ [6606945] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} time: 00:16:58 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606945/ [6606947] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:25:33 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606947/ [6606949] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:14:21 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606949/ [6606951] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:16:43 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606951/ [6606953] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:13:40 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606953/ [6606955] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} time: 00:21:51 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606955/ [6606956] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} time: 00:30:09 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606956/ [6606957] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/valgrind} time: 01:30:59 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606957/ [6606958] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:15:41 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606958/ [6606960] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} time: 00:45:58 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606960/ [6606961] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:05:58 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606961/ [6606962] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} time: 00:10:28 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606962/ [6606965] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:11:54 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606965/ [6606966] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:23:17 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606966/ [6606967] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} time: 00:20:54 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606967/ [6606968] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} time: 00:24:30 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606968/ [6606969] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:50 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606969/ [6606970] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:14:54 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606970/ [6606971] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:56 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606971/ [6606972] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606972/ [6606973] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:18:32 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606973/ [6606974] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:15:26 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606974/ [6606975] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} time: 01:27:55 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606975/ [6606976] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} time: 00:14:48 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606976/ [6606979] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:57:20 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606979/ [6606980] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:02:42 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606980/ [6606981] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:55:05 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606981/ [6606982] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} time: 00:44:59 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606982/ [6606984] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} time: 00:15:27 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606984/ [6606986] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:17:16 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606986/ [6606987] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:20:25 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606987/ [6606989] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:57 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606989/ [6606990] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:17:52 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606990/ [6606991] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:15:22 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606991/ [6606992] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush} time: 00:13:59 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606992/ [6606993] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:03:32 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606993/ [6606995] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:51:07 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606995/ [6606996] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 00:54:54 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606996/ [6606997] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} time: 00:21:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606997/ [6606999] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:11:17 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6606999/ [6607000] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:16:47 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6607000/ [6607001] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:18:34 info: http://pulpito.ceph.com/teuthology-2022-01-11_03:15:03-fs-master-distro-default-smithi/6607001/