2023-08-22T00:42:50.755 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-22T00:42:50.757 INFO:teuthology.results:subset: '111/120000' 2023-08-22T00:42:50.757 INFO:teuthology.results:seed: '2330' 2023-08-22T05:52:22.843 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-22T05:52:22.860 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-22T06:35:26.584 INFO:teuthology.results:Tests finished! gathering results... 2023-08-22T06:35:42.825 INFO:scrape:Found 177 jobs 2023-08-22T06:35:47.042 INFO:scrape:Found 8 distinct failure reasons 2023-08-22T06:35:47.043 INFO:scrape:Failure: Command failed on smithi078 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" 2023-08-22T06:35:47.043 INFO:scrape:2 jobs: ['7374977', '7375132'] 2023-08-22T06:35:47.044 INFO:scrape:suites intersection: ['0-kubeadm', '1-rook', '2-workload/radosbench', '3-final', 'cluster/1-node', 'k8s/1.21', 'net/calico'] 2023-08-22T06:35:47.044 INFO:scrape:suites union: ['0-kubeadm', '1-rook', '2-workload/radosbench', '3-final', 'cluster/1-node', 'k8s/1.21', 'net/calico', 'orch/rook/smoke/{0-distro/ubuntu_18.04', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.6.2}', 'rook/master}'] 2023-08-22T06:35:47.044 INFO:scrape: 2023-08-22T06:35:47.045 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=21b2d401852937440c3da8ce2b19224181760caa 2023-08-22T06:35:47.045 INFO:scrape:2 jobs: ['7375141', '7374917'] 2023-08-22T06:35:47.045 INFO:scrape:suites intersection: ['orch/cephadm/dashboard/{0-distro/ignorelist_health', 'task/test_e2e}'] 2023-08-22T06:35:47.045 INFO:scrape:suites union: ['orch/cephadm/dashboard/{0-distro/ignorelist_health', 'task/test_e2e}'] 2023-08-22T06:35:47.046 INFO:scrape: 2023-08-22T06:35:47.046 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2023-08-22T06:35:47.046 INFO:scrape:4 jobs: ['7375093', '7375225', '7374887', '7374973'] 2023-08-22T06:35:47.047 INFO:scrape:suites intersection: ['orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_nfs}'] 2023-08-22T06:35:47.047 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_nfs}'] 2023-08-22T06:35:47.047 INFO:scrape: 2023-08-22T06:35:47.048 INFO:scrape:Failure: Failed to reconnect to smithi196 2023-08-22T06:35:47.048 INFO:scrape:5 jobs: ['7374865', '7374986', '7374911', '7375222', '7375135'] 2023-08-22T06:35:47.048 INFO:scrape:suites intersection: [] 2023-08-22T06:35:47.048 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-install/mimic-v1only', '1-install/nautilus-v2only', '1-rook', '1-start', '2-ops/rmdir-reactivate}', '2-services/nfs-ingress-rgw-bucket', '2-workload/none', '3-final', '3-final}', 'backoff/normal', 'ceph', 'cluster/3-node', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'k8s/1.21', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'net/calico', 'orch/cephadm/osds/{0-distro/ubuntu_18.04', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'orch/rook/smoke/{0-distro/ubuntu_18.04', 'rados', 'rook/1.6.2}', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}', 'workloads/snaps-few-objects}'] 2023-08-22T06:35:47.049 INFO:scrape: 2023-08-22T06:35:47.049 INFO:scrape:Failure: Command failed on smithi175 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror'" 2023-08-22T06:35:47.049 INFO:scrape:1 jobs: ['7374877'] 2023-08-22T06:35:47.050 INFO:scrape:suites: ['1-install/octopus', 'backoff/peering', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/few', 'orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/rbd_cls}'] 2023-08-22T06:35:47.050 INFO:scrape: 2023-08-22T06:35:47.051 INFO:scrape:Failure: Command failed on smithi178 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b2d401852937440c3da8ce2b19224181760caa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 04a861d6-40ae-11ee-9b3a-001a4aab830c -- ceph mon dump -f json' 2023-08-22T06:35:47.052 INFO:scrape:1 jobs: ['7375198'] 2023-08-22T06:35:47.052 INFO:scrape:suites: ['1-start', '2-thrash', '3-tasks/rados_api_tests', 'fixed-2', 'msgr/async-v2only', 'orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'root}'] 2023-08-22T06:35:47.052 INFO:scrape: 2023-08-22T06:35:47.053 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=21b2d401852937440c3da8ce2b19224181760caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2023-08-22T06:35:47.053 INFO:scrape:2 jobs: ['7375001', '7374823'] 2023-08-22T06:35:47.053 INFO:scrape:suites intersection: ['orch/cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'task/test_e2e}'] 2023-08-22T06:35:47.054 INFO:scrape:suites union: ['orch/cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'task/test_e2e}'] 2023-08-22T06:35:47.054 INFO:scrape: 2023-08-22T06:35:47.054 INFO:scrape:Failure: Command failed on smithi052 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b2d401852937440c3da8ce2b19224181760caa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 385a6b94-40ab-11ee-9b3a-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\'' 2023-08-22T06:35:47.054 INFO:scrape:1 jobs: ['7375172'] 2023-08-22T06:35:47.055 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', 'orch/cephadm/osds/{0-distro/ubuntu_18.04'] 2023-08-22T06:35:47.055 INFO:scrape: 2023-08-22T06:35:47.065 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/ failed: 18 dead: 0 running: 0 waiting: 0 queued: 0 passed: 159 Fail ================================================================= [7374823] orch/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} ----------------------------------------------------------------- time: 00:26:22 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374823/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374823/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9f8034b2cc34984a27be32b34908eef Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi129 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=21b2d401852937440c3da8ce2b19224181760caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7374865] orch/rook/smoke/{0-distro/ubuntu_18.04 0-kubeadm 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/1.6.2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374865/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2287830baae04cd9a6c241c616e59405 Failed to reconnect to smithi196 [7374877] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:35:49 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374877/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374877/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5bbfd4e406d246fdb7f21d14dd1c194e Command failed on smithi175 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_cls_rbd --gtest_filter=-Te stClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror'" [7374887] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:24:59 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374887/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374887/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5481d804e81f482b93286db425bd1949 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7374911] orch/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374911/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374911/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c205cb182bdd4cb7bde68f5d366fc67b Failed to reconnect to smithi057 [7374917] orch/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} ----------------------------------------------------------------- time: 00:08:02 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374917/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374917/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba39f28e3f2d4b9b82d9be5692b3f18d Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =21b2d401852937440c3da8ce2b19224181760caa [7374973] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:25:40 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374973/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374973/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7703c2268155496bba8e90d53a16a986 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7374977] orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/master} ----------------------------------------------------------------- time: 00:06:13 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374977/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374977/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d2b35b9019f4659ac9900fcb8706b2f Command failed on smithi078 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7374986] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374986/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374986/ sentry: https://sentry.ceph.com/organizations/ceph/?query=afb09d82fdba47b4a89307120c1d9b9a Failed to reconnect to smithi175 [7375001] orch/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} ----------------------------------------------------------------- time: 00:26:40 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375001/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375001/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d21102d38e844f59c562b0b6f479b5f Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=21b2d401852937440c3da8ce2b19224181760caa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7375093] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:25:20 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375093/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375093/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad0bcf6243d0485c890e1f8465ffeaf1 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7375132] orch/rook/smoke/{0-distro/ubuntu_18.04 0-kubeadm 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.6.2} ----------------------------------------------------------------- time: 00:05:59 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375132/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375132/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58e5282f1f444f91a35e6c2c70feaeaa Command failed on smithi057 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7375135] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375135/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375135/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e2829ee974d44beb540ad0dc02a2a6e Failed to reconnect to smithi133 [7375141] orch/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} ----------------------------------------------------------------- time: 00:07:51 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375141/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375141/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b6958c3e47a4a8ca7201d1935026586 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =21b2d401852937440c3da8ce2b19224181760caa [7375172] orch/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:16:13 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375172/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375172/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6820681f0ace4c54b9358350b7b486f9 Command failed on smithi052 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:21b2d401852937440c3da8ce2b19224181760caa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 385a6b94-40ab-11ee-9b3a-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\'' [7375198] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:13:38 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375198/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375198/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ac52811a1eb49b299552a373abf2160 Command failed on smithi178 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:21b2d401852937440c3da8ce2b19224181760caa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 04a861d6-40ae-11ee-9b3a-001a4aab830c -- ceph mon dump -f json' [7375222] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375222/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375222/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d33732de0ad42f58a40e2a5e0101c29 Failed to reconnect to smithi184 [7375225] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:25:36 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375225/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375225/ sentry: https://sentry.ceph.com/organizations/ceph/?query=156ec6834f9141c6a2dc35ef1c8e581f Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) Pass ================================================================= [7374815] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:01 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374815/ [7374816] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:35:20 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374816/ [7374817] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:23:59 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374817/ [7374818] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli_mon} time: 00:25:37 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374818/ [7374819] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:18:53 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374819/ [7374821] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:13:46 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374821/ [7374825] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:54 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374825/ [7374827] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:30:23 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374827/ [7374829] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr orchestrator_cli} time: 00:21:00 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374829/ [7374831] orch/cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{centos_8} workloads/ceph_iscsi} time: 00:33:00 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374831/ [7374833] orch/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/classic start} time: 00:15:23 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374833/ [7374835] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:14:19 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374835/ [7374837] orch/cephadm/smoke-singlehost/{0-distro$/{ubuntu_18.04} 1-start 2-services/basic 3-final} time: 00:15:13 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374837/ [7374839] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} time: 00:31:19 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374839/ [7374841] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} time: 00:53:00 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374841/ [7374843] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:43:42 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374843/ [7374845] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_adoption} time: 00:13:07 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374845/ [7374847] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:44:55 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374847/ [7374849] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:19:43 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374849/ [7374851] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} time: 00:16:57 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374851/ [7374853] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/connectivity start} time: 00:20:17 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374853/ [7374855] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:27:03 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374855/ [7374857] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:28 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374857/ [7374859] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:39 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374859/ [7374861] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:18:39 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374861/ [7374863] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm_repos} time: 00:09:12 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374863/ [7374867] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} time: 00:37:19 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374867/ [7374869] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:30:04 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374869/ [7374871] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/classic start} time: 00:19:51 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374871/ [7374873] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:20:10 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374873/ [7374875] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:15:48 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374875/ [7374879] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:42 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374879/ [7374881] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:25:44 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374881/ [7374883] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:22:38 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374883/ [7374885] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:17:12 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374885/ [7374889] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:21:45 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374889/ [7374891] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:25:59 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374891/ [7374893] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:13:54 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374893/ [7374895] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} time: 00:35:19 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374895/ [7374897] orch/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:40:51 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374897/ [7374899] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:51:29 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374899/ [7374901] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} time: 00:15:15 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374901/ [7374903] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:17 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374903/ [7374905] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:23:21 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374905/ [7374907] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:17:24 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374907/ [7374909] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli_mon} time: 00:25:25 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374909/ [7374913] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:25:41 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374913/ [7374915] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:20:55 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374915/ [7374919] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:09 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374919/ [7374921] orch/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/connectivity start} time: 00:15:52 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374921/ [7374923] orch/cephadm/smoke-singlehost/{0-distro$/{rhel_8.4_container_tools_3.0} 1-start 2-services/rgw 3-final} time: 00:17:34 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374923/ [7374925] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 00:45:20 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374925/ [7374927] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} time: 00:52:20 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374927/ [7374929] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:31:08 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374929/ [7374931] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_adoption} time: 00:12:43 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374931/ [7374933] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:18:55 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374933/ [7374935] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:44:39 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374935/ [7374937] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} time: 00:16:50 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374937/ [7374939] orch/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/repave-all} time: 00:16:22 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374939/ [7374941] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/classic start} time: 00:19:43 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374941/ [7374943] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:17:17 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374943/ [7374945] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:51 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374945/ [7374947] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm_repos} time: 00:09:18 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374947/ [7374949] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:29:31 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374949/ [7374951] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:13:31 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374951/ [7374953] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:31:50 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374953/ [7374955] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:35:14 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374955/ [7374957] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:42:09 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374957/ [7374959] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/connectivity start} time: 00:17:43 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374959/ [7374961] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:15:26 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374961/ [7374963] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:17:52 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374963/ [7374965] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:35 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374965/ [7374967] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:32:18 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374967/ [7374969] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:47 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374969/ [7374971] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:35:46 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374971/ [7374975] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:21:52 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374975/ [7374979] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:16:23 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374979/ [7374980] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:36:26 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374980/ [7374982] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:29:15 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374982/ [7374984] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} time: 00:15:32 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374984/ [7374988] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:43 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374988/ [7374990] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:49:57 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374990/ [7374993] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:23:40 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374993/ [7374995] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:13:38 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374995/ [7374996] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli_mon} time: 00:25:22 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374996/ [7374999] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:20:16 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7374999/ [7375004] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:35 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375004/ [7375007] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:30:19 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375007/ [7375010] orch/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_3.0} 2-node-mgr orchestrator_cli} time: 00:18:56 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375010/ [7375012] orch/cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{centos_8} workloads/ceph_iscsi} time: 00:27:24 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375012/ [7375015] orch/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/classic start} time: 00:15:36 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375015/ [7375018] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:16:22 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375018/ [7375021] orch/cephadm/smoke-singlehost/{0-distro$/{ubuntu_18.04} 1-start 2-services/basic 3-final} time: 00:15:18 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375021/ [7375024] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:35:58 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375024/ [7375028] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} time: 00:25:13 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375028/ [7375031] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:38:47 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375031/ [7375035] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_adoption} time: 00:12:57 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375035/ [7375038] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:34:46 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375038/ [7375040] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:19:53 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375040/ [7375043] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} time: 00:17:33 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375043/ [7375046] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:16:51 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375046/ [7375049] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/connectivity start} time: 00:19:55 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375049/ [7375052] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:36:24 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375052/ [7375055] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:40 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375055/ [7375058] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:16:34 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375058/ [7375060] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm_repos} time: 00:09:19 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375060/ [7375063] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 00:44:49 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375063/ [7375066] orch/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:29:28 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375066/ [7375069] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/classic start} time: 00:17:55 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375069/ [7375072] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:16:55 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375072/ [7375075] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:16:45 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375075/ [7375078] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:32:29 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375078/ [7375081] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:16:43 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375081/ [7375084] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:52 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375084/ [7375087] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:53:30 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375087/ [7375090] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:23:18 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375090/ [7375096] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:21:45 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375096/ [7375099] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:35:44 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375099/ [7375102] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:26:03 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375102/ [7375105] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:15:46 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375105/ [7375108] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:33:06 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375108/ [7375111] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:41:55 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375111/ [7375114] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} time: 00:16:10 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375114/ [7375117] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:00 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375117/ [7375120] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:15:32 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375120/ [7375123] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:23:41 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375123/ [7375126] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:16:44 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375126/ [7375129] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli_mon} time: 00:25:53 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375129/ [7375138] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:16:07 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375138/ [7375145] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:24 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375145/ [7375148] orch/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/connectivity start} time: 00:16:09 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375148/ [7375151] orch/cephadm/smoke-singlehost/{0-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:17:48 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375151/ [7375154] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:39:03 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375154/ [7375157] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} time: 00:31:21 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375157/ [7375161] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:29:08 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375161/ [7375163] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_adoption} time: 00:15:16 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375163/ [7375166] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:17:15 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375166/ [7375169] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:33:53 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375169/ [7375175] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} time: 00:18:48 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375175/ [7375178] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/classic start} time: 00:19:48 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375178/ [7375181] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:19:05 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375181/ [7375184] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:44 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375184/ [7375186] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm_repos} time: 00:09:18 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375186/ [7375189] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:35:24 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375189/ [7375192] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:29:37 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375192/ [7375195] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:16:11 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375195/ [7375201] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:39:51 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375201/ [7375204] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/connectivity start} time: 00:18:09 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375204/ [7375207] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:16:18 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375207/ [7375210] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:17 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375210/ [7375213] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:18:47 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375213/ [7375216] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:02 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375216/ [7375219] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:44:51 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375219/ [7375228] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:21:47 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375228/ [7375230] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:18:18 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375230/ [7375233] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} time: 00:45:01 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375233/ [7375236] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:27:31 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375236/ [7375239] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} time: 00:15:47 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375239/ [7375242] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:21:32 info: http://pulpito.ceph.com/yuriw-2023-08-21_23:09:47-orch-pacific-release-distro-default-smithi/7375242/