2022-10-16T03:32:40.353 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-16T03:32:40.355 INFO:teuthology.results:subset: '15871/99999' 2022-10-16T03:32:40.355 INFO:teuthology.results:seed: '5330' 2022-10-16T09:17:04.146 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-16T09:17:04.180 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-16T11:47:19.783 INFO:teuthology.results:Tests finished! gathering results... 2022-10-16T11:47:55.968 INFO:scrape:Found 417 jobs 2022-10-16T11:47:58.771 INFO:scrape:Found 11 distinct failure reasons 2022-10-16T11:47:58.773 INFO:scrape:Failure: [Errno 2] Cannot find file on the remote 'ubuntu@smithi003.front.sepia.ceph.com': 'rook/cluster/examples/kubernetes/ceph/operator.yaml' 2022-10-16T11:47:58.774 INFO:scrape:2 jobs: ['7068309', '7068175'] 2022-10-16T11:47:58.775 INFO:scrape:suites intersection: ['0-kubeadm', '1-rook', '3-final', 'k8s/1.21', 'net/calico'] 2022-10-16T11:47:58.776 INFO:scrape:suites union: ['0-kubeadm', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'rados/rook/smoke/{0-distro/ubuntu_18.04', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.6.2}', 'rook/master}'] 2022-10-16T11:47:58.777 INFO:scrape: 2022-10-16T11:47:58.778 INFO:scrape:Failure: expected MON_CLOCK_SKEW but got none 2022-10-16T11:47:58.778 INFO:scrape:1 jobs: ['7068216'] 2022-10-16T11:47:58.779 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'no_pools', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/multimon/{clusters/3', 'supported-random-distro$/{rhel_8}', 'tasks/mon_clock_with_skews}'] 2022-10-16T11:47:58.780 INFO:scrape: 2022-10-16T11:47:58.781 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d3f4fdce2f201414613133139fc1397ad6453b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2022-10-16T11:47:58.782 INFO:scrape:1 jobs: ['7068505'] 2022-10-16T11:47:58.783 INFO:scrape:suites: ['rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'task/test_e2e}'] 2022-10-16T11:47:58.784 INFO:scrape: 2022-10-16T11:47:58.784 INFO:scrape:Failure: {'smithi080.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['hwclock', '--systohc'], 'delta': '0:00:00.002080', 'end': '2022-10-16 08:57:20.913301', 'invocation': {'module_args': {'_raw_params': 'hwclock --systohc', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-10-16 08:57:20.911221', 'stderr': 'hwclock: Cannot access the Hardware Clock via any known method.\nhwclock: Use the --verbose option to see the details of our search for an access method.', 'stderr_lines': ['hwclock: Cannot access the Hardware Clock via any known method.', 'hwclock: Use the --verbose option to see the details of our search for an access method.'], 'stdout': '', 'stdout_lines': []}} 2022-10-16T11:47:58.785 INFO:scrape:2 jobs: ['7068507', '7068529'] 2022-10-16T11:47:58.786 INFO:scrape:suites intersection: [] 2022-10-16T11:47:58.787 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fixed-2', 'mds', 'mode/root', 'mon', 'mon_election/connectivity', 'msgr/async', 'osd}', 'overrides/{pg-warn', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8', 'roles', 'start', 'tasks/rados_api_tests}', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-10-16T11:47:58.788 INFO:scrape: 2022-10-16T11:47:58.789 INFO:scrape:Failure: Cannot connect to remote host smithi099 2022-10-16T11:47:58.790 INFO:scrape:8 jobs: ['7068219', '7068393', '7068269', '7068296', '7068211', '7068189', '7068425', '7068492'] 2022-10-16T11:47:58.791 INFO:scrape:suites intersection: [] 2022-10-16T11:47:58.792 INFO:scrape:suites union: ['0-nvme-loop', '1-install/luminous-v1only', '1-install/mimic', '1-install/nautilus', '1-install/octopus', '1-start', '2-partial-upgrade/firsthalf', '2-services/basic', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress2', '3-final}', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-pacific', '8-workload/{rbd-python', 'backoff/peering', 'backoff/peering_and_degraded', 'bluestore-bitmap', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashers/careful', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'ubuntu_18.04}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}'] 2022-10-16T11:47:58.793 INFO:scrape: 2022-10-16T11:47:58.794 INFO:scrape:Failure: Command failed on smithi134 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 425a8a70-4d09-11ed-8437-001a4aab830c -e sha1=3d3f4fdce2f201414613133139fc1397ad6453b4 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2022-10-16T11:47:58.794 INFO:scrape:1 jobs: ['7068198'] 2022-10-16T11:47:58.795 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'roles', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-10-16T11:47:58.796 INFO:scrape: 2022-10-16T11:47:58.797 INFO:scrape:Failure: {'smithi080.front.sepia.ceph.com': {'changed': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi080.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}} 2022-10-16T11:47:58.798 INFO:scrape:1 jobs: ['7068506'] 2022-10-16T11:47:58.799 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{default}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/radosbench}'] 2022-10-16T11:47:58.800 INFO:scrape: 2022-10-16T11:47:58.801 INFO:scrape:Failure: Command failed on smithi047 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml' 2022-10-16T11:47:58.802 INFO:scrape:1 jobs: ['7068436'] 2022-10-16T11:47:58.803 INFO:scrape:suites: ['0-kubeadm', '1-rook', '2-workload/none', '3-final', 'cluster/1-node', 'k8s/1.21', 'net/calico', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.6.2}'] 2022-10-16T11:47:58.804 INFO:scrape: 2022-10-16T11:47:58.805 INFO:scrape:Failure: "2022-10-16T10:37:21.319575+0000 mon.a (mon.0) 6484 : cluster [ERR] scrub mismatch" in cluster log 2022-10-16T11:47:58.806 INFO:scrape:1 jobs: ['7068536'] 2022-10-16T11:47:58.807 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}'] 2022-10-16T11:47:58.808 INFO:scrape: 2022-10-16T11:47:58.809 INFO:scrape:Failure: Command failed on smithi111 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3d3f4fdce2f201414613133139fc1397ad6453b4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e709c246-4d1a-11ed-8437-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\'' 2022-10-16T11:47:58.810 INFO:scrape:1 jobs: ['7068360'] 2022-10-16T11:47:58.811 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', 'rados/cephadm/osds/{0-distro/ubuntu_18.04'] 2022-10-16T11:47:58.812 INFO:scrape: 2022-10-16T11:47:58.813 INFO:scrape:Failure: SSH connection to smithi080 was lost: 'sudo yum -y install ceph-radosgw' 2022-10-16T11:47:58.813 INFO:scrape:1 jobs: ['7068524'] 2022-10-16T11:47:58.814 INFO:scrape:suites: ['rados/objectstore/{backends/fusestore', 'supported-random-distro$/{centos_8}}'] 2022-10-16T11:47:58.815 INFO:scrape: 2022-10-16T11:47:58.837 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/ failed: 18 dead: 2 running: 0 waiting: 0 queued: 0 passed: 397 Fail ================================================================= [7068175] rados/rook/smoke/{0-distro/ubuntu_18.04 0-kubeadm 1-rook 2-workload/radosbench 3-final cluster/3-node k8s/1.21 net/calico rook/1.6.2} ----------------------------------------------------------------- time: 00:05:26 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068175/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068175/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e974c2b1891d4ebeb95c59173b8df3ce Command failed on smithi005 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml' [7068189] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068189/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fdd54b24d6874e3fb04ee35a62a7e9f2 Cannot connect to remote host smithi170 [7068198] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:18:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068198/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068198/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c585a56c6174b8c85e5ea62d1869a9b Command failed on smithi134 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 425a8a70-4d09-11ed-8437-001a4aab830c -e sha1=3d3f4fdce2f201414613133139fc1397ad6453b4 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7068211] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068211/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068211/ sentry: https://sentry.ceph.com/organizations/ceph/?query=970c9a06d6dc486eb9837facda8ba946 Cannot connect to remote host smithi142 [7068216] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} ----------------------------------------------------------------- time: 00:08:25 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068216/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068216/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41b7f9d4fe604cdda475e4d3e972dd52 expected MON_CLOCK_SKEW but got none [7068219] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068219/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068219/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59ac6a7758f247319d2f859ae383a605 Cannot connect to remote host smithi099 [7068269] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068269/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068269/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b8814a472ae4103b84838ed70274a32 Cannot connect to remote host smithi160 [7068296] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-pacific 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068296/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068296/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2054aef081e44fb28bc3e18e0d31b3f2 Cannot connect to remote host smithi097 [7068309] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 1-rook 2-workload/none 3-final cluster/1-node k8s/1.21 net/calico rook/master} ----------------------------------------------------------------- time: 00:04:21 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068309/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068309/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89d4de85dd274731973e35c9fdbd5744 [Errno 2] Cannot find file on the remote 'ubuntu@smithi003.front.sepia.ceph.com': 'rook/cluster/examples/kubernetes/ceph/operator.yaml' [7068360] rados/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:14:19 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068360/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068360/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9437893c0a24e45af70899747e8f4b9 Command failed on smithi111 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:3d3f4fdce2f201414613133139fc1397ad6453b4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e709c246-4d1a-11ed-8437-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\'' [7068393] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068393/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068393/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9bf2e31895b14587825dc297dbff410d Cannot connect to remote host smithi090 [7068425] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068425/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068425/ sentry: https://sentry.ceph.com/organizations/ceph/?query=431f6ca20a4d4e1981616e7982cdc5c0 Cannot connect to remote host smithi061 [7068436] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 1-rook 2-workload/none 3-final cluster/1-node k8s/1.21 net/calico rook/1.6.2} ----------------------------------------------------------------- time: 00:04:35 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068436/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068436/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5acac1acc564f7eaa49db28f23c1a5e Command failed on smithi047 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml' [7068492] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068492/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068492/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c04fe6127c74368a03556621fd124bf Cannot connect to remote host smithi049 [7068505] rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} ----------------------------------------------------------------- time: 00:19:21 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068505/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068505/ sentry: https://sentry.ceph.com/organizations/ceph/?query=37fcf0031a474067859912ae12db3ba1 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d3f4fdce2f201414613133139fc1397ad6453b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7068506] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:04:37 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068506/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=732ec2a7060a49ef9f7dba31c1bc8e86 {'smithi080.front.sepia.ceph.com': {'changed': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi080.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}} [7068524] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:04:02 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068524/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068524/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1230c8f5ea394bf9ba07f9d214115104 SSH connection to smithi080 was lost: 'sudo yum -y install ceph-radosgw' [7068536] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 02:10:32 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068536/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068536/ "2022-10-16T10:37:21.319575+0000 mon.a (mon.0) 6484 : cluster [ERR] scrub mismatch" in cluster log Dead ================================================================= [7068507] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:03:08 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068507/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068507/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0cece37e5b6e441896dbcdbb03895295 {'smithi080.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['hwclock', '--systohc'], 'delta': '0:00:00.002080', 'end': '2022-10-16 08:57:20.913301', 'invocation': {'module_args': {'_raw_params': 'hwclock --systohc', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-10-16 08:57:20.911221', 'stderr': 'hwclock: Cannot access the Hardware Clock via any known method.\nhwclock: Use the --verbose option to see the details of our search for an access method.', 'stderr_lines': ['hwclock: Cannot access the Hardware Clock via any known method.', 'hwclock: Use the --verbose option to see the details of our search for an access method.'], 'stdout': '', 'stdout_lines': []}} [7068529] rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:03:57 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068529/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068529/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ee31f58189e4307af8901dbb543fb15 {'smithi080.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['rpm', '-e', 'katello-ca-consumer- satellite.front.sepia.ceph.com-1.0-1.noarch'], 'delta': '0:00:00.006788', 'end': '2022-10-16 09:10:39.547172', 'invocation': {'module_args': {'_raw_params': 'rpm -e "katello-ca-consumer- satellite.front.sepia.ceph.com-1.0-1.noarch"', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-10-16 09:10:39.540384', 'stderr': 'error: package katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-1.noarch is not installed', 'stderr_lines': ['error: package katello-ca-consumer- satellite.front.sepia.ceph.com-1.0-1.noarch is not installed'], 'stdout': '', 'stdout_lines': []}} Pass ================================================================= [7068135] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:28 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068135/ [7068136] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:09 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068136/ [7068137] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-radosbench} time: 01:39:09 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068137/ [7068138] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm_repos} time: 00:05:01 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068138/ [7068139] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 00:54:34 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068139/ [7068140] rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:26:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068140/ [7068141] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/rados_api_tests} time: 00:26:59 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068141/ [7068142] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 01:39:47 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068142/ [7068143] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:09:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068143/ [7068144] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:44:21 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068144/ [7068145] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:05:51 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068145/ [7068146] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/classic start} time: 00:14:52 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068146/ [7068147] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:16:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068147/ [7068148] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:01 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068148/ [7068149] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:34 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068149/ [7068150] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:10:11 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068150/ [7068151] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:11:08 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068151/ [7068152] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_8}} time: 00:13:49 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068152/ [7068153] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:41:57 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068153/ [7068154] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} time: 00:24:32 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068154/ [7068155] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_5925} time: 00:11:29 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068155/ [7068156] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:49:05 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068156/ [7068157] rados/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/repave-all} time: 00:15:12 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068157/ [7068158] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:10:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068158/ [7068159] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:16:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068159/ [7068160] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:50:20 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068160/ [7068161] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/workunits} time: 00:11:23 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068161/ [7068162] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:12:35 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068162/ [7068163] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:20:20 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068163/ [7068164] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:11:59 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068164/ [7068165] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:41:19 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068165/ [7068166] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:26:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068166/ [7068167] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:20 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068167/ [7068168] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench} time: 00:25:59 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068168/ [7068169] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} time: 00:13:04 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068169/ [7068170] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:10:51 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068170/ [7068171] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:38:23 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068171/ [7068172] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:11:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068172/ [7068173] rados/standalone/{supported-random-distro$/{centos_8} workloads/osd} time: 03:11:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068173/ [7068174] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:13:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068174/ [7068176] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:32:06 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068176/ [7068177] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:10:05 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068177/ [7068178] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:24 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068178/ [7068179] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:21:12 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068179/ [7068180] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} time: 00:22:03 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068180/ [7068181] rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:35:38 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068181/ [7068182] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:33:27 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068182/ [7068183] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068183/ [7068184] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:09:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068184/ [7068185] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:21:19 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068185/ [7068186] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:09:15 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068186/ [7068187] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:16:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068187/ [7068188] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:14:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068188/ [7068190] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:13:29 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068190/ [7068191] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli_mon} time: 00:24:20 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068191/ [7068192] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:19 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068192/ [7068193] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:16:32 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068193/ [7068194] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068194/ [7068195] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/none msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:08:39 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068195/ [7068196] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:12:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068196/ [7068197] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_api_tests} time: 00:29:15 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068197/ [7068199] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:12:24 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068199/ [7068200] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:28:42 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068200/ [7068201] rados/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/connectivity start} time: 00:14:17 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068201/ [7068202] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:08:15 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068202/ [7068203] rados/cephadm/smoke-singlehost/{0-distro$/{ubuntu_18.04} 1-start 2-services/rgw 3-final} time: 00:13:56 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068203/ [7068204] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/crash} time: 00:12:14 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068204/ [7068205] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_adoption} time: 00:08:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068205/ [7068206] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:11:33 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068206/ [7068207] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} time: 00:53:32 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068207/ [7068208] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:44 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068208/ [7068209] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:36:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068209/ [7068210] rados/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python} time: 00:27:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068210/ [7068212] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:10:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068212/ [7068213] rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} time: 00:43:13 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068213/ [7068214] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:09:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068214/ [7068215] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/scrub_test} time: 00:09:15 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068215/ [7068217] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068217/ [7068218] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:20:59 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068218/ [7068220] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read} time: 00:22:02 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068220/ [7068221] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:15:08 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068221/ [7068222] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} time: 00:13:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068222/ [7068223] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/classic start} time: 00:15:42 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068223/ [7068224] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:11:19 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068224/ [7068225] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:40:11 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068225/ [7068226] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:17:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068226/ [7068227] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:15:16 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068227/ [7068228] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:07:39 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068228/ [7068229] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:27 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068229/ [7068230] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:18:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068230/ [7068231] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:57 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068231/ [7068232] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:33:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068232/ [7068233] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} time: 01:41:04 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068233/ [7068234] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:01 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068234/ [7068235] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:29:26 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068235/ [7068236] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:44:14 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068236/ [7068237] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm_repos} time: 00:04:45 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068237/ [7068238] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_mon_osdmap_prune} time: 00:39:56 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068238/ [7068239] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:14:05 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068239/ [7068240] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:34:29 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068240/ [7068241] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:19:32 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068241/ [7068242] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:19:53 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068242/ [7068243] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:10:32 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068243/ [7068244] rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:39:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068244/ [7068245] rados/objectstore/{backends/objectstore supported-random-distro$/{rhel_8}} time: 04:44:46 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068245/ [7068246] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_cls_all validater/lockdep} time: 00:13:38 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068246/ [7068247] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/connectivity start} time: 00:14:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068247/ [7068248] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:22:17 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068248/ [7068249] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:15:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068249/ [7068250] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:47:44 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068250/ [7068251] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:15:52 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068251/ [7068252] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:11 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068252/ [7068253] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:16 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068253/ [7068254] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} time: 00:23:47 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068254/ [7068255] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:15:58 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068255/ [7068256] rados/singleton/{all/peer mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:05:59 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068256/ [7068257] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:06:03 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068257/ [7068258] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:10:13 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068258/ [7068259] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:51 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068259/ [7068260] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:36 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068260/ [7068261] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:13:35 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068261/ [7068262] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:18:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068262/ [7068263] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:29:27 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068263/ [7068264] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:20:04 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068264/ [7068265] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:11:47 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068265/ [7068266] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:28:22 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068266/ [7068267] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:13:20 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068267/ [7068268] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} time: 00:11:58 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068268/ [7068270] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:22:24 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068270/ [7068271] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:51 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068271/ [7068272] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 00:47:11 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068272/ [7068273] rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:22:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068273/ [7068274] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:38 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068274/ [7068275] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:30:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068275/ [7068276] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:16:33 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068276/ [7068277] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_mon_workunits} time: 00:28:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068277/ [7068278] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:13 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068278/ [7068279] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/rados_api_tests} time: 00:26:44 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068279/ [7068280] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:14:25 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068280/ [7068281] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:22:29 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068281/ [7068282] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/insights} time: 00:13:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068282/ [7068283] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli_mon} time: 00:23:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068283/ [7068284] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:59 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068284/ [7068285] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:06:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068285/ [7068286] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_cls_all validater/valgrind} time: 02:08:33 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068286/ [7068287] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:12:51 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068287/ [7068288] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-overwrites} time: 00:23:53 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068288/ [7068289] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/none msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:09:07 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068289/ [7068290] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:21:32 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068290/ [7068291] rados/dashboard/{centos_8.stream_container_tools clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_8} tasks/dashboard} time: 01:13:15 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068291/ [7068292] rados/objectstore/{backends/alloc-hint supported-random-distro$/{ubuntu_latest}} time: 00:06:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068292/ [7068293] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:09:49 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068293/ [7068294] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:18:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068294/ [7068295] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:09:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068295/ [7068297] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:21:01 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068297/ [7068298] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:23:38 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068298/ [7068299] rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} time: 00:20:45 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068299/ [7068300] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:52 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068300/ [7068301] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:08:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068301/ [7068302] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068302/ [7068303] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:58 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068303/ [7068304] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:11:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068304/ [7068305] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects} time: 00:21:18 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068305/ [7068306] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:01 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068306/ [7068307] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:27:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068307/ [7068308] rados/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04} 2-node-mgr orchestrator_cli} time: 00:10:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068308/ [7068310] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:49 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068310/ [7068311] rados/cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{rhel_8} workloads/ceph_iscsi} time: 00:24:45 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068311/ [7068312] rados/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/classic start} time: 00:13:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068312/ [7068313] rados/cephadm/smoke-singlehost/{0-distro$/{ubuntu_18.04} 1-start 2-services/basic 3-final} time: 00:12:46 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068313/ [7068314] rados/singleton/{all/random-eio mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:20:08 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068314/ [7068315] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/rados_cls_all} time: 00:15:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068315/ [7068316] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_adoption} time: 00:09:03 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068316/ [7068317] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:16 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068317/ [7068318] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:26:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068318/ [7068319] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:06:04 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068319/ [7068320] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:13:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068320/ [7068321] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:10:25 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068321/ [7068322] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:30:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068322/ [7068323] rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:35:42 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068323/ [7068324] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/snaps-few-objects} time: 00:28:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068324/ [7068325] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:15:09 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068325/ [7068326] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} time: 00:50:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068326/ [7068327] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:50:42 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068327/ [7068328] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/module_selftest} time: 00:31:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068328/ [7068329] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:11:11 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068329/ [7068330] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/mon_recovery validater/lockdep} time: 00:10:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068330/ [7068331] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:01 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068331/ [7068332] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:58 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068332/ [7068333] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:12:53 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068333/ [7068334] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 01:04:02 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068334/ [7068335] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} time: 00:13:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068335/ [7068336] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/connectivity start} time: 00:16:19 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068336/ [7068337] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:15 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068337/ [7068338] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{rhel_8}} time: 00:13:13 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068338/ [7068339] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:12:22 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068339/ [7068340] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:06:52 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068340/ [7068341] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:05:59 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068341/ [7068342] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:29:57 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068342/ [7068343] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:46 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068343/ [7068344] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:33:27 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068344/ [7068345] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm_repos} time: 00:04:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068345/ [7068346] rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:09:35 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068346/ [7068347] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/rados_python} time: 00:13:11 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068347/ [7068348] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:11:26 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068348/ [7068349] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:05:56 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068349/ [7068350] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:32:25 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068350/ [7068351] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:35:15 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068351/ [7068352] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:25:25 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068352/ [7068353] rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:24:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068353/ [7068354] rados/standalone/{supported-random-distro$/{centos_8} workloads/erasure-code} time: 00:40:36 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068354/ [7068355] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/classic start} time: 00:14:25 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068355/ [7068356] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:07:19 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068356/ [7068357] rados/singleton/{all/test-crash mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:07:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068357/ [7068358] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:45 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068358/ [7068359] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:16:04 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068359/ [7068361] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:16:34 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068361/ [7068362] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068362/ [7068363] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/many workloads/pool-create-delete} time: 00:19:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068363/ [7068364] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} time: 00:23:39 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068364/ [7068365] rados/singleton/{all/test-noautoscale-flag mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:07:59 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068365/ [7068366] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:16:05 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068366/ [7068367] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:07:24 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068367/ [7068368] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:48:53 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068368/ [7068369] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:20:22 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068369/ [7068370] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/valgrind} time: 00:52:33 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068370/ [7068371] rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:23:07 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068371/ [7068372] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:19:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068372/ [7068373] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:06:08 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068373/ [7068374] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:10:11 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068374/ [7068375] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:14:35 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068375/ [7068376] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:26:15 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068376/ [7068377] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:14:28 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068377/ [7068378] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} time: 00:12:21 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068378/ [7068379] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:57 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068379/ [7068380] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:31:07 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068380/ [7068381] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:12:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068381/ [7068382] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:32:38 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068382/ [7068383] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:10:33 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068383/ [7068384] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:02 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068384/ [7068385] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:33 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068385/ [7068386] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:24:47 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068386/ [7068387] rados/objectstore/{backends/filejournal supported-random-distro$/{centos_8}} time: 00:07:07 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068387/ [7068388] rados/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:38:51 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068388/ [7068389] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 01:05:39 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068389/ [7068390] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068390/ [7068391] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:21:58 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068391/ [7068392] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:07:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068392/ [7068394] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:21:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068394/ [7068395] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:07 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068395/ [7068396] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068396/ [7068397] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:31:23 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068397/ [7068398] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:15:34 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068398/ [7068399] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli_mon} time: 00:23:52 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068399/ [7068400] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:01 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068400/ [7068401] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:15:02 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068401/ [7068402] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:20:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068402/ [7068403] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:27:12 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068403/ [7068404] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925} time: 00:06:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068404/ [7068405] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:06 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068405/ [7068406] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:26:57 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068406/ [7068407] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:18:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068407/ [7068408] rados/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/connectivity start} time: 00:14:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068408/ [7068409] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:06:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068409/ [7068410] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:10:04 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068410/ [7068411] rados/cephadm/smoke-singlehost/{0-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:15:19 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068411/ [7068412] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_cls_all validater/lockdep} time: 00:15:07 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068412/ [7068413] rados/standalone/{supported-random-distro$/{rhel_8} workloads/mgr} time: 00:12:17 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068413/ [7068414] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_adoption} time: 00:08:38 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068414/ [7068415] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:11:46 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068415/ [7068416] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:34:56 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068416/ [7068417] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} time: 00:51:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068417/ [7068418] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:21:08 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068418/ [7068419] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:23:58 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068419/ [7068420] rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python} time: 00:28:12 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068420/ [7068421] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} time: 00:52:36 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068421/ [7068422] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:08:46 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068422/ [7068423] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:23:57 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068423/ [7068424] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:26 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068424/ [7068426] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:22:18 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068426/ [7068427] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:14:52 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068427/ [7068428] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:10:50 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068428/ [7068429] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:05:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068429/ [7068430] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} time: 00:13:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068430/ [7068431] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{ubuntu_latest}} time: 02:19:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068431/ [7068432] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:20:23 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068432/ [7068433] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_fio} time: 00:07:56 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068433/ [7068434] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/classic start} time: 00:22:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068434/ [7068435] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:18:41 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068435/ [7068437] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:39:08 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068437/ [7068438] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:35 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068438/ [7068439] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:15:28 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068439/ [7068440] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:05:44 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068440/ [7068441] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:18:07 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068441/ [7068442] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_big} time: 00:28:50 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068442/ [7068443] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068443/ [7068444] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:30:24 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068444/ [7068445] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm_repos} time: 00:04:49 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068445/ [7068446] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/rados_api_tests} time: 00:26:14 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068446/ [7068447] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache} time: 00:11:14 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068447/ [7068448] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:11:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068448/ [7068449] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} time: 00:30:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068449/ [7068450] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:21:53 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068450/ [7068451] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:06:39 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068451/ [7068452] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:19:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068452/ [7068453] rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:32:45 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068453/ [7068454] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind} time: 00:28:59 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068454/ [7068455] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/connectivity start} time: 00:14:38 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068455/ [7068456] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:32:07 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068456/ [7068457] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:14:03 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068457/ [7068458] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:07:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068458/ [7068459] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:06:04 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068459/ [7068460] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} time: 00:32:47 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068460/ [7068461] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:17:27 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068461/ [7068462] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:07:45 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068462/ [7068463] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068463/ [7068464] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:25:39 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068464/ [7068465] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:08:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068465/ [7068466] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:44 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068466/ [7068467] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:21:44 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068467/ [7068468] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} time: 00:23:02 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068468/ [7068469] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-radosbench} time: 02:15:02 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068469/ [7068470] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:30:00 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068470/ [7068471] rados/standalone/{supported-random-distro$/{rhel_8} workloads/misc} time: 00:30:13 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068471/ [7068472] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:09:58 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068472/ [7068473] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix} time: 00:17:36 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068473/ [7068474] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:20:51 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068474/ [7068475] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:14:05 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068475/ [7068476] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068476/ [7068477] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rados_api_tests} time: 00:26:29 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068477/ [7068478] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:27:42 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068478/ [7068479] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:42:18 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068479/ [7068480] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{ubuntu_latest}} time: 02:26:03 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068480/ [7068481] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} time: 00:11:56 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068481/ [7068482] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:28:54 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068482/ [7068483] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:06:38 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068483/ [7068484] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} time: 00:37:06 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068484/ [7068485] rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:24:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068485/ [7068486] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068486/ [7068487] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_mon_osdmap_prune} time: 00:42:20 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068487/ [7068488] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:11:12 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068488/ [7068489] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} time: 00:12:39 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068489/ [7068490] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:07:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068490/ [7068491] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} time: 00:59:57 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068491/ [7068493] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:07:26 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068493/ [7068494] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/crash} time: 00:12:09 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068494/ [7068495] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/lockdep} time: 00:22:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068495/ [7068496] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:21:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068496/ [7068497] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli_mon} time: 00:23:35 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068497/ [7068498] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:31:14 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068498/ [7068499] rados/dashboard/{centos_8.stream_container_tools clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:44 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068499/ [7068500] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:17:05 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068500/ [7068501] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-pacific 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_18.04} time: 02:48:16 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068501/ [7068502] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:15:40 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068502/ [7068503] rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 01:11:53 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068503/ [7068504] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:27:55 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068504/ [7068508] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:07:14 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068508/ [7068509] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:05 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068509/ [7068510] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:29 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068510/ [7068511] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:27:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068511/ [7068512] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-balanced} time: 00:20:53 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068512/ [7068513] rados/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_3.0} 2-node-mgr orchestrator_cli} time: 00:14:16 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068513/ [7068514] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:09:57 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068514/ [7068515] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 01:25:11 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068515/ [7068516] rados/cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{rhel_8} workloads/ceph_iscsi} time: 00:24:25 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068516/ [7068517] rados/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/classic start} time: 00:13:46 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068517/ [7068518] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068518/ [7068519] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:07:36 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068519/ [7068520] rados/cephadm/smoke-singlehost/{0-distro$/{ubuntu_18.04} 1-start 2-services/basic 3-final} time: 00:13:05 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068520/ [7068521] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect} time: 00:13:03 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068521/ [7068522] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_adoption} time: 00:08:26 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068522/ [7068523] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:11:21 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068523/ [7068525] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:06:45 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068525/ [7068526] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:23:22 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068526/ [7068527] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:11:32 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068527/ [7068528] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} time: 00:38:43 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068528/ [7068530] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:25:27 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068530/ [7068531] rados/standalone/{supported-random-distro$/{rhel_8} workloads/mon} time: 00:43:35 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068531/ [7068532] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:12:10 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068532/ [7068533] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:34:16 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068533/ [7068534] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/failover} time: 00:12:30 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068534/ [7068535] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} time: 00:13:15 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068535/ [7068537] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 01:25:20 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068537/ [7068538] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/readwrite} time: 00:11:44 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068538/ [7068539] rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} time: 00:23:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068539/ [7068540] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:48 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068540/ [7068541] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:14:08 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068541/ [7068542] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} time: 00:13:51 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068542/ [7068543] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/connectivity start} time: 00:16:07 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068543/ [7068544] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 01:20:32 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068544/ [7068545] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:37 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068545/ [7068546] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:07:52 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068546/ [7068547] rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:13:23 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068547/ [7068548] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:08:16 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068548/ [7068549] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:13:18 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068549/ [7068550] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:18 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068550/ [7068551] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:31 info: http://pulpito.ceph.com/teuthology-2022-10-16_03:31:03-rados-pacific-distro-default-smithi/7068551/