2022-10-25T11:15:21.283 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-25T11:15:21.284 INFO:teuthology.results:subset: '9148/100000' 2022-10-25T11:15:21.285 INFO:teuthology.results:seed: '7496' 2022-10-25T12:19:55.625 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-25T12:19:55.653 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-26T00:20:55.307 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-10-26T00:21:04.933 INFO:scrape:Found 131 jobs 2022-10-26T00:21:14.852 INFO:scrape:Found 6 distinct failure reasons 2022-10-26T00:21:14.853 INFO:scrape:Failure: Command failed on smithi027 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' 2022-10-26T00:21:14.854 INFO:scrape:92 jobs 2022-10-26T00:21:14.855 INFO:scrape:suites intersection: [] 2022-10-26T00:21:14.855 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/mirror', '2-services/nfs', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs2', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch/cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0', 'orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8', 'orch/cephadm/osds/{0-distro/ubuntu_20.04', 'orch/cephadm/rbd_iscsi/{0-single-container-host', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools}', 'orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_3.0}', 'orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_rhel8}', 'orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04}', 'orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'orch/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0', 'orch/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8', 'orch/cephadm/smoke/{0-distro/ubuntu_20.04', 'orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'orch/cephadm/thrash/{0-distro/rhel_8.4_container_tools_3.0', 'orch/cephadm/thrash/{0-distro/rhel_8.4_container_tools_rhel8', 'orch/cephadm/thrash/{0-distro/ubuntu_20.04', 'orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0', 'orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8', 'orch/cephadm/with-work/{0-distro/ubuntu_20.04', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0', 'orch/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8', 'orch/cephadm/workunits/{0-distro/ubuntu_20.04', 'root}', 'start', 'start}', 'task/test_nfs}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'workloads/cephadm_iscsi}'] 2022-10-26T00:21:14.856 INFO:scrape: 2022-10-26T00:21:14.856 INFO:scrape:Failure: Command failed on smithi049 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 293a8468-545d-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 2\'"\'"\'\'' 2022-10-26T00:21:14.857 INFO:scrape:14 jobs: ['7078335', '7078383', '7078292', '7078282', '7078400', '7078318', '7078341', '7078315', '7078303', '7078368', '7078347', '7078405', '7078359', '7078382'] 2022-10-26T00:21:14.857 INFO:scrape:suites intersection: [] 2022-10-26T00:21:14.858 INFO:scrape:suites union: ['1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-bootstrap/octopus', '1-start', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9'] 2022-10-26T00:21:14.858 INFO:scrape: 2022-10-26T00:21:14.859 INFO:scrape:Dead: b'2022-10-25T23:26:50.248 DEBUG:teuthology.exit:Finished running handlers' 2022-10-26T00:21:14.859 INFO:scrape:10 jobs: ['7078289', '7078337', '7078376', '7078324', '7078351', '7078401', '7078313', '7078367', '7078305', '7078387'] 2022-10-26T00:21:14.859 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2022-10-26T00:21:14.860 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4'] 2022-10-26T00:21:14.860 INFO:scrape: 2022-10-26T00:21:14.861 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi197 with status 125: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5fbb8171010367eef3729cd64d88a414f76e7d7a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2022-10-26T00:21:14.861 INFO:scrape:2 jobs: ['7078358', '7078308'] 2022-10-26T00:21:14.862 INFO:scrape:suites intersection: ['task/test_cephadm}'] 2022-10-26T00:21:14.862 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'task/test_cephadm}'] 2022-10-26T00:21:14.863 INFO:scrape: 2022-10-26T00:21:14.863 INFO:scrape:Failure: 'wait for toolbox' reached maximum tries (100) after waiting for 500 seconds 2022-10-26T00:21:14.863 INFO:scrape:2 jobs: ['7078300', '7078366'] 2022-10-26T00:21:14.864 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '3-final', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}'] 2022-10-26T00:21:14.864 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}'] 2022-10-26T00:21:14.865 INFO:scrape: 2022-10-26T00:21:14.865 INFO:scrape:Failure: Command failed on smithi079 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml' 2022-10-26T00:21:14.866 INFO:scrape:2 jobs: ['7078399', '7078333'] 2022-10-26T00:21:14.866 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/radosbench', '3-final', 'cluster/3-node', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}'] 2022-10-26T00:21:14.867 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/radosbench', '3-final', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}'] 2022-10-26T00:21:14.867 INFO:scrape: 2022-10-26T00:21:14.871 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/ failed: 112 dead: 9 running: 1 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [7078280] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:07:02 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078280/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078280/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a429017e043641f299afcd6107064e74 Command failed on smithi093 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078281] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:05:17 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078281/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078281/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5e2404d13b84f4fb49af6c9e38c168a Command failed on smithi018 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078282] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:27 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078282/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078282/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c08a87678024d1889326a227c76c252 Command failed on smithi145 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ca9bf1e-5458-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078283] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:08:16 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078283/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078283/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8c788f996c74399aa80e8a046c9c8a3 Command failed on smithi153 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078284] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:09:58 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078284/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078284/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3eed3562c5974d5cad055e8c6333c8d6 Command failed on smithi059 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078285] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:09:56 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078285/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078285/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2da8da432dd54620b6ef9b0f11452c54 Command failed on smithi027 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078286] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:06:50 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078286/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078286/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f6fcf2870494a7e90a655c8c8e101e2 Command failed on smithi143 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078287] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:09:44 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078287/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078287/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3692535950f4f6abe21d7ef2297011a Command failed on smithi044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078288] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:07:40 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078288/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078288/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1f03d2f8ed94a3bb964b8d51eb93777 Command failed on smithi061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078290] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:06:44 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078290/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078290/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e63fd1b239ab4447805efe485ea39b16 Command failed on smithi094 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078291] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:04:52 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078291/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078291/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5db220fbc14e48a0a2d1931ac4542e01 Command failed on smithi071 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078292] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:20:46 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078292/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078292/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5fe08d22baed4b2aacb9b9ea3269e585 Command failed on smithi035 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 008ef9f8-5459-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078294] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:06:29 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078294/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078294/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16f4764511da452caeac8ef086527019 Command failed on smithi078 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078295] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:06:40 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078295/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078295/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f77253d65e4d439dac240158b03dcbd0 Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078296] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:06:30 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078296/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078296/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9aec5ecba9284e74885dcdcb229c3e7f Command failed on smithi154 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078297] orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:05:20 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078297/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078297/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bd66f24fce141cca7607f7266016fe2 Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078298] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:09:41 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078298/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078298/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1695bd4aa6d840db961961ee2a1db54b Command failed on smithi112 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078299] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:09:42 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078299/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078299/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d9d6391399f4732940064a6d4c2f721 Command failed on smithi093 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078300] orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/host rook/master} ----------------------------------------------------------------- time: 00:19:21 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078300/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078300/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a93c006100c4427584c57bbc5256724b 'wait for toolbox' reached maximum tries (100) after waiting for 500 seconds [7078302] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:54 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078302/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078302/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1cbec80eab754f36a1644f859a7bfd37 Command failed on smithi049 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078303] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:20:42 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078303/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078303/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc8d3e598fb44ccfbdf72ba965a80b3b Command failed on smithi062 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b9836fc4-545a-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)"\'' [7078304] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:08:03 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078304/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078304/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dbe487cf4abc4337a9b9ed5e1eefb9cb Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078306] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:09:14 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078306/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078306/ sentry: https://sentry.ceph.com/organizations/ceph/?query=566e55f3619c4312b3ce193cc32ef0ac Command failed on smithi153 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078307] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:08:08 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078307/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078307/ sentry: https://sentry.ceph.com/organizations/ceph/?query=505dd5d00e7f478ca3b16b7f4de825d4 Command failed on smithi050 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078308] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:11:07 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078308/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078308/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be4159f887e3432891a42ca223f00fb0 Command failed (workunit test cephadm/test_cephadm.sh) on smithi156 with status 125: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5fbb8171010367eef3729cd64d88a414f76e7d7a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7078309] orch/cephadm/thrash/{0-distro/rhel_8.4_container_tools_3.0 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:12:51 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078309/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078309/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8a2f726c472a49ecbf1cb052daf2549d Command failed on smithi102 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078310] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:13:07 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078310/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078310/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b347e94dae54e8f9429192dda1a9a67 Command failed on smithi018 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078311] orch/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:09:17 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078311/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078311/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33faa0b2e61540f88ac360f2b87e7bb1 Command failed on smithi143 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078312] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:08:51 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078312/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078312/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5a211377a4944619956ac5a19823eee Command failed on smithi189 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078314] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:06:48 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078314/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078314/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71dfc51fb6ab46d992f8fc4aab845ede Command failed on smithi026 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078315] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:18:20 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078315/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078315/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b04a047e2aa0435385e7cebfc2b967de Command failed on smithi036 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cc10196c-545a-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078316] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:09:19 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078316/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078316/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ff37b9f50af44eb8186d62aa13c28db Command failed on smithi094 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078318] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:18:11 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078318/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078318/ sentry: https://sentry.ceph.com/organizations/ceph/?query=773aa673d7354f6682b1439c6eda47df Command failed on smithi061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08b4593c-545b-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078319] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:08:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078319/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078319/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e158285bfe37413cbc3edbf2cc42058f Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078320] orch/cephadm/thrash/{0-distro/rhel_8.4_container_tools_rhel8 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:12:43 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078320/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078320/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88ffa83bfe944e17b8679c6a020fa843 Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078321] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:12:16 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078321/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078321/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f69149b3d50f428b830748ab5cd68dab Command failed on smithi071 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078322] orch/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:08:35 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078322/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078322/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64feef05aa4b4bf9bde12899ac94de24 Command failed on smithi167 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078323] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:07:52 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078323/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078323/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d03d45c56404cc9a418d27bdd3ba326 Command failed on smithi044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078325] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:08:59 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078325/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078325/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ebb1339a0f84428ac8ad9222664d5bd Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078326] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:08:19 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078326/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078326/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33bf87926cef4e0b91d1cfbaa79614b6 Command failed on smithi103 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078327] orch/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/off mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:11:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078327/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b313dab099234e2c8c495ffb5342a2d9 Command failed on smithi196 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078328] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:04:48 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078328/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5980a2d8aa174ce782dabe77888edf7a Command failed on smithi045 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078330] orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:07:17 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078330/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078330/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e18779470e54535b49331cf54ba2827 Command failed on smithi191 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078331] orch/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:06:56 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078331/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078331/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62f22c07b923499cab992436a6c77017 Command failed on smithi052 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078332] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:07:29 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078332/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078332/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2925d79a73ca4f139348547cf2344090 Command failed on smithi079 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078333] orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/3-node k8s/1.21 net/calico rook/1.7.2} ----------------------------------------------------------------- time: 00:09:36 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078333/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078333/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7b85f69a55543d587af83abf844c7f4 Command failed on smithi112 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml' [7078334] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:07:58 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078334/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078334/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0034ac21f3a441b2b834bab094dd3471 Command failed on smithi093 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078335] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:24:57 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078335/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078335/ sentry: https://sentry.ceph.com/organizations/ceph/?query=195c2623eb9e472c92f89abbb902c62f Command failed on smithi049 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 293a8468-545d-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 2\'"\'"\'\'' [7078336] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:06:48 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078336/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f68fabd4cada4310ba241aa85e94a48b Command failed on smithi145 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078338] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:42 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078338/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078338/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd1b7f4e610c4902aeebcf1c68aaa996 Command failed on smithi033 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078339] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:07:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078339/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078339/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d05babca5e61498c906bda3656c55e3b Command failed on smithi050 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078340] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:07:55 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078340/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078340/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1bd04d680974555a9f7a23b6500843e Command failed on smithi043 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078341] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:20:57 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078341/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078341/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ef4cfbedeca431a878260ab56846cd8 Command failed on smithi153 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f7ce31a-545c-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078342] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:07:40 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078342/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dbeb4035bdfa4653bc9e98dc4d7ae365 Command failed on smithi143 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078343] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:10:57 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078343/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70b456a7fd22431ea83e96693cd680ed Command failed on smithi189 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078344] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:11:19 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078344/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078344/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce0f3c4e12754b58bcebea5959509725 Command failed on smithi035 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078345] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:08:21 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078345/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078345/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16260d39e05e444cba97c692b6145312 Command failed on smithi094 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078346] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:08:26 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078346/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078346/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65a82931d33c4f0db24b35782a5bacc2 Command failed on smithi157 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078347] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:20:28 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078347/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078347/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1375cccc3eb04754befdb6e5f149d66c Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e67468a-545d-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078348] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:07:03 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078348/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078348/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fb4ec3a041943c49f0de3a255c52698 Command failed on smithi026 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078349] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:09:43 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078349/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8fc8c9a41b414cc3b3e486a9bfc1484b Command failed on smithi005 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078352] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:11:24 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078352/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078352/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aaf0b40ac9a24eea9fe22a9f101dfecf Command failed on smithi102 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078353] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:11:13 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078353/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078353/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0309b2d3a6fc4d7681c27bd61ace2fe2 Command failed on smithi027 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078354] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:09:12 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078354/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078354/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4b24f4941c9a4e8897be43dae7ea521a Command failed on smithi044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078355] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:05:51 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078355/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078355/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c3607b921104913b52b72369cd6febc Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078356] orch/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:09:36 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078356/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=17098cb332b443d68ffca3508d079f9a Command failed on smithi167 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078357] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:08:15 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078357/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078357/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d7bcd9844c0c49e5a71cff758e4b82fd Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078358] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:11:16 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078358/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078358/ sentry: https://sentry.ceph.com/organizations/ceph/?query=562eb6bef59d4f67bbfbd978e232e505 Command failed (workunit test cephadm/test_cephadm.sh) on smithi197 with status 125: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5fbb8171010367eef3729cd64d88a414f76e7d7a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7078359] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:22:09 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078359/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078359/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3c77bd5651141dab2a7710ab5c51c02 Command failed on smithi085 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b6c416c-545d-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078361] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:07:48 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078361/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078361/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55578f752cb4413cac4cd32010fa1c1e Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078362] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:07:18 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078362/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078362/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ebcfe6b06e84c1f9334b751688fec35 Command failed on smithi045 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078363] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:06:56 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078363/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078363/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e539a16759bc4e0489c89c4a8e993a10 Command failed on smithi084 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078364] orch/cephadm/thrash/{0-distro/rhel_8.4_container_tools_3.0 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:11:47 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078364/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078364/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6c00e55699345dc84f6fe366a367ed8 Command failed on smithi071 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078365] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:11:38 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078365/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078365/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50d679504328491d8c2788ed600e9c92 Command failed on smithi036 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078366] orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/1-node k8s/1.21 net/flannel rook/master} ----------------------------------------------------------------- time: 00:21:30 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078366/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078366/ sentry: https://sentry.ceph.com/organizations/ceph/?query=99cd105f1cb44fee80c65fe589005d1f 'wait for operator' reached maximum tries (90) after waiting for 900 seconds [7078368] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:15:59 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078368/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078368/ sentry: https://sentry.ceph.com/organizations/ceph/?query=110250f663db4d39844955f0bdc1f94d Command failed on smithi052 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ddfb5cc-545e-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)"\'' [7078369] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:06:38 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078369/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078369/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f08f687cf21a4f439a76daa8e8772c0e Command failed on smithi061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078370] orch/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:07:25 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078370/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078370/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93b4b9a80925481f90d0764b0b980639 Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078371] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:06:36 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078371/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078371/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29e968fa77464506bbfc3f56b1484835 Command failed on smithi093 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078373] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:08:08 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078373/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078373/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8f37cd8e3bd41f08980611c6420e158 Command failed on smithi033 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078374] orch/cephadm/thrash/{0-distro/rhel_8.4_container_tools_rhel8 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:10:54 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078374/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078374/ sentry: https://sentry.ceph.com/organizations/ceph/?query=372efd523fa7445281095a74a3d06d45 Command failed on smithi143 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078375] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:11:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078375/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078375/ sentry: https://sentry.ceph.com/organizations/ceph/?query=713d230836734f309f4ddb5d26ba4d1f Command failed on smithi062 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078377] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:08:01 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078377/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078377/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db16de80b7474062a58079c53e7d5b09 Command failed on smithi043 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078378] orch/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:05:43 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078378/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078378/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c64d4468bf194b828d21f76ac73e75c4 Command failed on smithi050 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078379] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:05:46 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078379/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078379/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56f299b066dd497894776e00fa9b7fce Command failed on smithi094 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078380] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:08:16 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078380/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd21bd8ff419447ba96d35ec33d9299e Command failed on smithi132 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078381] orch/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/on mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:11:26 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078381/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=11e01ea6385846058a5c9accb6a288b4 Command failed on smithi112 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078382] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:59 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078382/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078382/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e8ab7a9cb2a4b3c8968f82cb97267b7 Command failed on smithi157 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 899749fc-545f-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078383] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:16:42 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078383/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=269e297cdfd741048c6af1a10adf87eb Command failed on smithi189 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1de1a27a-545f-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078384] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:07:13 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078384/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078384/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9bb83be35644293a2dd89c347d6f546 Command failed on smithi044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078385] orch/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:07:55 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078385/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078385/ sentry: https://sentry.ceph.com/organizations/ceph/?query=91f09d2a3e1e439992aec8bd384d58cb Command failed on smithi026 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078386] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:07:34 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078386/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078386/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ebfb2c4c2894226900b8c8558dd8dd0 Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078388] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:07:13 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078388/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f59588326264670a053396d968dc8e9 Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078389] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:07:26 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078389/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078389/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f64751c2a0a424fb0425c40959f1f6a Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078390] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:08:05 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078390/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7cd125e8c2a74fce826b7ff6202f7f56 Command failed on smithi035 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078391] orch/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/off mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:11:01 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078391/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078391/ sentry: https://sentry.ceph.com/organizations/ceph/?query=24352cdef84e4c44b483b9a2698de2af Command failed on smithi084 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078392] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:08:33 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078392/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078392/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34436961f06d4c13be511eaf01109edb Command failed on smithi167 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078394] orch/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_3.0} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:08:26 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078394/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078394/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61aa5357c5c04fe3b367f4ba4a99a78a Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078395] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:10:08 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078395/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eca32c7c8d094efbb1bb67fd285a3df3 Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078396] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:10:09 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078396/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078396/ sentry: https://sentry.ceph.com/organizations/ceph/?query=065ee3ba1a374a99822af2ff702b1cf3 Command failed on smithi053 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078397] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:07:46 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078397/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=37e642f6be0d49e08b68a59f31f4f15a Command failed on smithi102 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078398] orch/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:01 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078398/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078398/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ed58c0e21e848a9ae8f85243b5115f7 Command failed on smithi045 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078399] orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/3-node k8s/1.21 net/host rook/1.7.2} ----------------------------------------------------------------- time: 00:07:36 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078399/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078399/ sentry: https://sentry.ceph.com/organizations/ceph/?query=536a6bbf4d614619bfe1be12cec50951 Command failed on smithi079 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml' [7078400] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:18:31 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078400/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078400/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f800a6f59ab445859fc09081b04f9386 Command failed on smithi083 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7b1d18b0-5460-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)"\'' [7078402] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:07:20 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078402/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04043727dc1d4f7894168fe6a0d7fb36 Command failed on smithi071 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078403] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:05:15 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078403/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078403/ sentry: https://sentry.ceph.com/organizations/ceph/?query=90d9af951ed84dfd81f577308baa9f1c Command failed on smithi061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078405] orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:20:02 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078405/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078405/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e001c7d0287c40a3aa2bd0f830d0f782 Command failed on smithi036 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7f61fd2-5460-11ed-8438-001a4aab830c -e sha1=5219abe5bdb882abcc3a550aa02e563f2cd638bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7078406] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:06:53 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078406/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078406/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ede504b94c34fff9cdbda2edd264941 Command failed on smithi103 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078407] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:09:45 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078407/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078407/ sentry: https://sentry.ceph.com/organizations/ceph/?query=25ddbd9b7416441a965c8c1f50564e56 Command failed on smithi143 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078408] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:09:39 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078408/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e33eb8cdf2944072b5f4b419baaaf405 Command failed on smithi087 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078409] orch/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:08:22 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078409/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078409/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8725dbe434a34d8f979bacefca18e26b Command failed on smithi043 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' [7078410] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:06:55 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078410/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078410/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b11ce3cc310142c3a9e4c6a1f6f41a5e Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:5219abe5bdb882abcc3a550aa02e563f2cd638bd pull' Dead ================================================================= [7078289] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078289/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078289/ hit max job timeout [7078305] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078305/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078305/ hit max job timeout [7078313] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078313/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078313/ hit max job timeout [7078324] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078324/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078324/ hit max job timeout [7078337] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078337/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078337/ hit max job timeout [7078351] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078351/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078351/ hit max job timeout [7078367] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078367/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078367/ hit max job timeout [7078376] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078376/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078376/ hit max job timeout [7078387] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078387/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078387/ hit max job timeout Running ================================================================= [7078401] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078401/ Pass ================================================================= [7078293] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/off orchestrator_cli} time: 00:19:59 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078293/ [7078301] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/classic task/test_adoption} time: 00:11:44 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078301/ [7078317] orch/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/on mon_election/classic task/test_cephadm_repos} time: 00:10:20 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078317/ [7078329] orch/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 2-node-mgr agent/on orchestrator_cli} time: 00:21:12 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078329/ [7078350] orch/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/connectivity task/test_adoption} time: 00:10:57 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078350/ [7078360] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/off orchestrator_cli} time: 00:17:59 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078360/ [7078372] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/connectivity task/test_cephadm_repos} time: 00:07:49 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078372/ [7078393] orch/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 2-node-mgr agent/on orchestrator_cli} time: 00:18:57 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078393/ [7078404] orch/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/classic task/test_adoption} time: 00:12:51 info: http://pulpito.ceph.com/teuthology-2022-10-23_07:07:02-orch-quincy-distro-default-smithi/7078404/