2022-10-22T06:58:46.316 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-22T06:58:46.318 INFO:teuthology.results:subset: '111/120000' 2022-10-22T06:58:46.318 INFO:teuthology.results:seed: '5965' 2022-10-22T08:44:04.848 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-22T08:44:04.876 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-22T09:10:07.378 INFO:teuthology.results:Tests finished! gathering results... 2022-10-22T09:10:18.766 INFO:scrape:Found 173 jobs 2022-10-22T09:10:38.582 INFO:scrape:Found 7 distinct failure reasons 2022-10-22T09:10:38.584 INFO:scrape:Failure: Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-33d47140-51e2-11ed-8438-001a4aab830c@mon.a' 2022-10-22T09:10:38.585 INFO:scrape:122 jobs 2022-10-22T09:10:38.586 INFO:scrape:suites intersection: [] 2022-10-22T09:10:38.587 INFO:scrape:suites union: ['0-nvme-loop', '1-install/luminous', '1-install/luminous-v1only', '1-install/mimic', '1-install/mimic-v1only', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-install/octopus', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/mirror', '2-services/nfs', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs2', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'distro/centos_8.stream_container_tools', 'distro/rhel_8.4_container_tools_3.0', 'distro/rhel_8.4_container_tools_rhel8', 'distro/ubuntu_18.04', 'distro/ubuntu_20.04', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch/cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0', 'orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8', 'orch/cephadm/osds/{0-distro/ubuntu_18.04', 'orch/cephadm/osds/{0-distro/ubuntu_20.04', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'orch/cephadm/smoke-singlehost/{0-distro$/{centos_8.stream_container_tools}', 'orch/cephadm/smoke-singlehost/{0-distro$/{rhel_8.4_container_tools_3.0}', 'orch/cephadm/smoke-singlehost/{0-distro$/{rhel_8.4_container_tools_rhel8}', 'orch/cephadm/smoke-singlehost/{0-distro$/{ubuntu_18.04}', 'orch/cephadm/smoke/{0-nvme-loop', 'orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0', 'orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8', 'orch/cephadm/with-work/{0-distro/ubuntu_18.04', 'orch/cephadm/with-work/{0-distro/ubuntu_20.04', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'rados', 'root}', 'start', 'start}', 'task/test_e2e}', 'task/test_nfs}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] 2022-10-22T09:10:38.587 INFO:scrape: 2022-10-22T09:10:38.588 INFO:scrape:Failure: Command failed on smithi133 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7262f9a6-51e0-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2022-10-22T09:10:38.589 INFO:scrape:30 jobs 2022-10-22T09:10:38.590 INFO:scrape:suites intersection: [] 2022-10-22T09:10:38.591 INFO:scrape:suites union: ['1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-bootstrap/octopus', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-verify}', '4-verify}}', '4-wait', '5-upgrade-ls', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'mon_election/connectivity}', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4'] 2022-10-22T09:10:38.592 INFO:scrape: 2022-10-22T09:10:38.592 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi145 with status 125: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2022-10-22T09:10:38.593 INFO:scrape:4 jobs: ['7076808', '7076766', '7076720', '7076678'] 2022-10-22T09:10:38.593 INFO:scrape:suites intersection: ['orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] 2022-10-22T09:10:38.593 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] 2022-10-22T09:10:38.594 INFO:scrape: 2022-10-22T09:10:38.595 INFO:scrape:Failure: [Errno 2] Cannot find file on the remote 'ubuntu@smithi170.front.sepia.ceph.com': 'rook/cluster/examples/kubernetes/ceph/operator.yaml' 2022-10-22T09:10:38.595 INFO:scrape:2 jobs: ['7076739', '7076794'] 2022-10-22T09:10:38.595 INFO:scrape:suites intersection: ['0-kubeadm', '1-rook', '2-workload/radosbench', '3-final', 'cluster/1-node', 'k8s/1.21', 'net/calico'] 2022-10-22T09:10:38.596 INFO:scrape:suites union: ['0-kubeadm', '1-rook', '2-workload/radosbench', '3-final', 'cluster/1-node', 'k8s/1.21', 'net/calico', 'orch/rook/smoke/{0-distro/ubuntu_18.04', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.6.2}', 'rook/master}'] 2022-10-22T09:10:38.596 INFO:scrape: 2022-10-22T09:10:38.597 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi035 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2022-10-22T09:10:38.597 INFO:scrape:4 jobs: ['7076729', '7076687', '7076817', '7076773'] 2022-10-22T09:10:38.598 INFO:scrape:suites intersection: ['orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm_repos}'] 2022-10-22T09:10:38.598 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm_repos}'] 2022-10-22T09:10:38.599 INFO:scrape: 2022-10-22T09:10:38.600 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a 2022-10-22T09:10:38.600 INFO:scrape:2 jobs: ['7076710', '7076797'] 2022-10-22T09:10:38.601 INFO:scrape:suites intersection: ['orch/cephadm/dashboard/{0-distro/ignorelist_health', 'task/test_e2e}'] 2022-10-22T09:10:38.601 INFO:scrape:suites union: ['orch/cephadm/dashboard/{0-distro/ignorelist_health', 'task/test_e2e}'] 2022-10-22T09:10:38.602 INFO:scrape: 2022-10-22T09:10:38.602 INFO:scrape:Failure: Command failed on smithi074 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml' 2022-10-22T09:10:38.603 INFO:scrape:1 jobs: ['7076685'] 2022-10-22T09:10:38.603 INFO:scrape:suites: ['0-kubeadm', '1-rook', '2-workload/none', '3-final', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'orch/rook/smoke/{0-distro/ubuntu_18.04', 'rook/1.6.2}'] 2022-10-22T09:10:38.604 INFO:scrape: 2022-10-22T09:10:38.610 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/ failed: 165 dead: 0 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7076656] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:08:43 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076656/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076656/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2f146eb3b56485fb035b603d03ceaec Command failed on smithi201 with status 5: 'sudo systemctl stop ceph-07ffb8a8-51d9-11ed-8438-001a4aab830c@mon.a' [7076657] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:04:23 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076657/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076657/ sentry: https://sentry.ceph.com/organizations/ceph/?query=171e095c87914126b477f5f796049402 Command failed on smithi154 with status 5: 'sudo systemctl stop ceph-7a22a590-51d8-11ed-8438-001a4aab830c@mon.smithi154' [7076658] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:14 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076658/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076658/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3aec99b659924b19820fb4c10bf22f9b Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 509ee5de-51d9-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076659] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:05:52 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076659/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076659/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ad8bf43bcdf4250b44c113a86211899 Command failed on smithi074 with status 5: 'sudo systemctl stop ceph-521f1424-51d9-11ed-8438-001a4aab830c@mon.a' [7076660] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:04:42 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076660/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076660/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ee92e8e10e24126ab43ecdd7a581910 Command failed on smithi008 with status 5: 'sudo systemctl stop ceph-b90b5350-51d9-11ed-8438-001a4aab830c@mon.a' [7076661] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:09:44 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076661/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076661/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d962446203004ddb82738164774682a7 Command failed on smithi035 with status 5: 'sudo systemctl stop ceph-8f0f8d04-51da-11ed-8438-001a4aab830c@mon.a' [7076662] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:04:46 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076662/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076662/ sentry: https://sentry.ceph.com/organizations/ceph/?query=327e7a17c8914cd1a5883736613a5c2c Command failed on smithi040 with status 5: 'sudo systemctl stop ceph-6f4dc7ce-51da-11ed-8438-001a4aab830c@mon.smithi040' [7076663] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:06:20 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076663/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076663/ sentry: https://sentry.ceph.com/organizations/ceph/?query=05d7876ed85c4e81a4b6bce3a5c5bda2 Command failed on smithi154 with status 5: 'sudo systemctl stop ceph-5dc910ee-51da-11ed-8438-001a4aab830c@mon.smithi154' [7076664] orch/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} ----------------------------------------------------------------- time: 00:09:12 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076664/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076664/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93c4102d367848429e8b9edfed10ab50 Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-e1c196f0-51da-11ed-8438-001a4aab830c@mon.a' [7076665] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:55 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076665/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076665/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9017686322c343409a7616c68044e9e7 Command failed on smithi036 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 17d373b2-51db-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076666] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:21:11 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076666/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076666/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a1ecadb984346eb9ed5b075561d1727 Command failed on smithi003 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdab373e-51da-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076669] orch/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:06:21 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076669/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076669/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f64d320ac0064cf6999627bd4edc78d8 Command failed on smithi167 with status 5: 'sudo systemctl stop ceph-452698f8-51db-11ed-8438-001a4aab830c@mon.a' [7076670] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:06:19 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076670/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076670/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20bdf4ca82eb4e1cb47349542f530dd4 Command failed on smithi170 with status 5: 'sudo systemctl stop ceph-3efd0264-51db-11ed-8438-001a4aab830c@mon.smithi170' [7076671] orch/cephadm/smoke-singlehost/{0-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:06:12 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076671/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076671/ sentry: https://sentry.ceph.com/organizations/ceph/?query=886a3950099c4f61b9e966f8d568c22a Command failed on smithi074 with status 5: 'sudo systemctl stop ceph-393860e4-51db-11ed-8438-001a4aab830c@mon.smithi074' [7076672] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:09:09 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076672/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076672/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea1a357e2d51405fa2774e2a30faa902 Command failed on smithi103 with status 5: 'sudo systemctl stop ceph- ef13e2a8-51db-11ed-8438-001a4aab830c@mon.a' [7076673] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} ----------------------------------------------------------------- time: 00:19:12 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076673/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076673/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e48015103cac46019bb32ff00fbbb351 Command failed on smithi008 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18c9b52-51db-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)"\'' [7076674] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:10:50 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076674/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076674/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3cbc6b82f7142e6b0a12d45f0d35331 Command failed on smithi191 with status 5: 'sudo systemctl stop ceph-6e0b9b00-51dc-11ed-8438-001a4aab830c@mon.a' [7076676] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:06:05 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076676/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076676/ sentry: https://sentry.ceph.com/organizations/ceph/?query=99216572e157488fa0c8d59b32ab2f90 Command failed on smithi087 with status 5: 'sudo systemctl stop ceph-f00beb6a-51db-11ed-8438-001a4aab830c@mon.a' [7076677] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:07:41 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076677/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076677/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7898b1dfebb04132bcc643ba5ce4e826 Command failed on smithi154 with status 5: 'sudo systemctl stop ceph-5c580f7e-51dc-11ed-8438-001a4aab830c@mon.smithi154' [7076678] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:09:55 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076678/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076678/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8612b1ed74c47e1a74167058f88dea8 Command failed (workunit test cephadm/test_cephadm.sh) on smithi046 with status 125: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7076679] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:07:50 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076679/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076679/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc37a43343a34d55ab7226310f140127 Command failed on smithi040 with status 5: 'sudo systemctl stop ceph-82f68444-51dc-11ed-8438-001a4aab830c@mon.a' [7076680] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:06:10 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076680/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076680/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39a02e265153419cbc70785f153efd70 Command failed on smithi066 with status 5: 'sudo systemctl stop ceph-97b14b30-51dc-11ed-8438-001a4aab830c@mon.a' [7076681] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:07:58 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076681/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076681/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aecc173de676467eaccbd913c6797192 Command failed on smithi035 with status 5: 'sudo systemctl stop ceph-a3e700de-51dc-11ed-8438-001a4aab830c@mon.smithi035' [7076682] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:44 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076682/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076682/ sentry: https://sentry.ceph.com/organizations/ceph/?query=712ac3ba7e0f49fca46c81106efbfa67 Command failed on smithi133 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6352c76-51dc-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076683] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:07:23 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076683/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076683/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04159f9f724b4b0eb61b5a11127d7b60 Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-a6099f52-51dc-11ed-8438-001a4aab830c@mon.smithi043' [7076684] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:09:06 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076684/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076684/ sentry: https://sentry.ceph.com/organizations/ceph/?query=855951bf67ca45079a4d479f0ad69a50 Command failed on smithi079 with status 5: 'sudo systemctl stop ceph-1c76ab9e-51dd-11ed-8438-001a4aab830c@mon.a' [7076685] orch/rook/smoke/{0-distro/ubuntu_18.04 0-kubeadm 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/1.6.2} ----------------------------------------------------------------- time: 00:06:52 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076685/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076685/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee3d7ca2a0c140bd8485181bf5cff4cf Command failed on smithi074 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml' [7076686] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:10:21 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076686/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076686/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5c0fe166a9e4b9da8727c5452ed6973 Command failed on smithi167 with status 5: 'sudo systemctl stop ceph-a12914b2-51dd-11ed-8438-001a4aab830c@mon.a' [7076687] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:07:02 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076687/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076687/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f2d15a9d24f4c86b61334a1432352b8 Command failed (workunit test cephadm/test_repos.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7076688] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:17 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076688/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076688/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6bfc9495f59e462ca76c7c35ca35727a Command failed on smithi057 with status 5: 'sudo systemctl stop ceph-5cd55096-51dd-11ed-8438-001a4aab830c@mon.a' [7076689] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:04:29 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076689/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076689/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c43ca6c59e0d49d8ab31a3731a183042 Command failed on smithi103 with status 5: 'sudo systemctl stop ceph-6b0ea8ec-51dd-11ed-8438-001a4aab830c@mon.smithi103' [7076690] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:06:17 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076690/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076690/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1b461df60aa42999da1adb65577b51a Command failed on smithi017 with status 5: 'sudo systemctl stop ceph- ca23d7e4-51dd-11ed-8438-001a4aab830c@mon.a' [7076691] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:21:35 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076691/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076691/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0760125fc9524b12945a37ba31233298 Command failed on smithi018 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36fa3688-51de-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076692] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} ----------------------------------------------------------------- time: 00:16:42 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076692/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076692/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e629513b768a41779af5ffc01c037f71 Command failed on smithi160 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid debd04a0-51dd-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076693] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:08:54 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076693/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076693/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8344049f84274e749713218ba5cacb42 Command failed on smithi087 with status 5: 'sudo systemctl stop ceph-2e5a7506-51de-11ed-8438-001a4aab830c@mon.a' [7076694] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:05:00 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076694/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076694/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39480d6ab801449fa869b21f2c861853 Command failed on smithi085 with status 5: 'sudo systemctl stop ceph-55200836-51de-11ed-8438-001a4aab830c@mon.smithi085' [7076695] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:07:20 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076695/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076695/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8371c2b9345b49a7809583eb291e477c Command failed on smithi191 with status 5: 'sudo systemctl stop ceph-4be3dc70-51de-11ed-8438-001a4aab830c@mon.smithi191' [7076696] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:04:48 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076696/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076696/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01cc06f6bf7a4970a1c6cdc75c5a298e Command failed on smithi154 with status 5: 'sudo systemctl stop ceph-34796366-51de-11ed-8438-001a4aab830c@mon.a' [7076697] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:16:51 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076697/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076697/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfea315a5dbb4163b9437a91e71cd986 Command failed on smithi043 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6108f662-51de-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076698] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:06:44 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076698/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076698/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02b1e3522fa149c8a22d9ece6e90a494 Command failed on smithi036 with status 5: 'sudo systemctl stop ceph-88d1e78a-51de-11ed-8438-001a4aab830c@mon.smithi036' [7076699] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:09:27 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076699/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076699/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6eba8d784e2449aacd52a4afefbd5a1 Command failed on smithi040 with status 5: 'sudo systemctl stop ceph-e2724050-51de-11ed-8438-001a4aab830c@mon.a' [7076700] orch/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:06:37 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076700/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076700/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4efc293a8dfd4e18888b7f8358ee84f7 Command failed on smithi145 with status 5: 'sudo systemctl stop ceph-8ad5657a-51de-11ed-8438-001a4aab830c@mon.a' [7076701] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:09:28 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076701/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076701/ sentry: https://sentry.ceph.com/organizations/ceph/?query=44796e8b3fda4b658d891ff82a564f40 Command failed on smithi066 with status 5: 'sudo systemctl stop ceph-1612a1c0-51df-11ed-8438-001a4aab830c@mon.a' [7076702] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:06:10 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076702/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076702/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52bc3fbf943f4551a84aa438c10acde5 Command failed on smithi035 with status 5: 'sudo systemctl stop ceph-85d0c204-51de-11ed-8438-001a4aab830c@mon.a' [7076703] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:41 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076703/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076703/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5891a5156f84599ade6cc2704023bde Command failed on smithi003 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid faa7be0c-51de-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076704] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:04:56 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076704/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ef28fa1f1d4417c92371a7078c108f3 Command failed on smithi078 with status 5: 'sudo systemctl stop ceph-f5b5964e-51de-11ed-8438-001a4aab830c@mon.a' [7076705] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:08:13 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076705/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076705/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0396ea53d07433ab51e668aa7cfc2ef Command failed on smithi012 with status 5: 'sudo systemctl stop ceph-09078716-51df-11ed-8438-001a4aab830c@mon.smithi012' [7076706] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:10:06 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076706/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076706/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e16e3c4274a430b96e6a859961a427e Command failed on smithi074 with status 5: 'sudo systemctl stop ceph-74da3f74-51df-11ed-8438-001a4aab830c@mon.a' [7076707] orch/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:04:55 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076707/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076707/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e96a6b6d621d4163a0c6a73349fccf3e Command failed on smithi159 with status 5: 'sudo systemctl stop ceph- eb711974-51de-11ed-8438-001a4aab830c@mon.smithi159' [7076708] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:06:05 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076708/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076708/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a431e01f92e240e8ac4f6acc2736e1a2 Command failed on smithi057 with status 5: 'sudo systemctl stop ceph-1d337a42-51df-11ed-8438-001a4aab830c@mon.a' [7076709] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:07:21 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076709/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076709/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22e43dc5f54b4dffba710262082d77cb Command failed on smithi103 with status 5: 'sudo systemctl stop ceph-67fcd712-51df-11ed-8438-001a4aab830c@mon.smithi103' [7076710] orch/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} ----------------------------------------------------------------- time: 00:05:28 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076710/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076710/ sentry: https://sentry.ceph.com/organizations/ceph/?query=798e87e3bcb34d01b54c1c61d6ad4c11 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a [7076711] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:20 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076711/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076711/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bec0a50a0d124580bb5a432ecf2a4a17 Command failed on smithi167 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6662b80-51df-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076712] orch/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:06:40 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076712/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076712/ sentry: https://sentry.ceph.com/organizations/ceph/?query=965ca7d23e0b46f5985ad9c3c7d19ae8 Command failed on smithi037 with status 5: 'sudo systemctl stop ceph-b5c457f4-51df-11ed-8438-001a4aab830c@mon.a' [7076713] orch/cephadm/smoke-singlehost/{0-distro$/{ubuntu_18.04} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:04:30 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076713/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076713/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b67afc39d00a4679b4ce942f9bd5ab71 Command failed on smithi017 with status 5: 'sudo systemctl stop ceph-5c766322-51df-11ed-8438-001a4aab830c@mon.smithi017' [7076714] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:09:40 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076714/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076714/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0fe1c10ea14437482a5eaf8783430e4 Command failed on smithi008 with status 5: 'sudo systemctl stop ceph-3e83de20-51e0-11ed-8438-001a4aab830c@mon.a' [7076715] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} ----------------------------------------------------------------- time: 00:23:19 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076715/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076715/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee69de3c16c04326abbb768e5ad1d4c1 Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3faee038-51e0-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 2\'"\'"\'\'' [7076716] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:07:11 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076716/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076716/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78727b8401394177ab283ffd5631701b Command failed on smithi191 with status 5: 'sudo systemctl stop ceph-96a8b21a-51e0-11ed-8438-001a4aab830c@mon.a' [7076718] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:05:12 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076718/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076718/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19287f4377bf49f38ad3cef4921ed367 Command failed on smithi084 with status 5: 'sudo systemctl stop ceph-f63df5d8-51df-11ed-8438-001a4aab830c@mon.smithi084' [7076719] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:06:33 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076719/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076719/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c288ab815c3d42798665640bade526c6 Command failed on smithi045 with status 5: 'sudo systemctl stop ceph-37493416-51e0-11ed-8438-001a4aab830c@mon.a' [7076720] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:10:14 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076720/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076720/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51e744750784491d9751ad87dd5b2519 Command failed (workunit test cephadm/test_cephadm.sh) on smithi080 with status 125: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7076721] orch/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:05:17 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076721/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076721/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1da85fe661484e8db4f06d57080297d8 Command failed on smithi085 with status 5: 'sudo systemctl stop ceph-155b1a2c-51e0-11ed-8438-001a4aab830c@mon.smithi085' [7076722] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:08:23 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076722/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076722/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b62fc6b413f8455caabf95c5bd0ba0d5 Command failed on smithi059 with status 5: 'sudo systemctl stop ceph-89f7321c-51e0-11ed-8438-001a4aab830c@mon.a' [7076723] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:05:22 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076723/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076723/ sentry: https://sentry.ceph.com/organizations/ceph/?query=192d4ac99737429f91702024d56fa73c Command failed on smithi006 with status 5: 'sudo systemctl stop ceph-a6d7277a-51e0-11ed-8438-001a4aab830c@mon.smithi006' [7076724] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:21:45 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076724/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076724/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a6980a0916d4d34b221fd23d11cfe7c Command failed on smithi036 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce763000-51e0-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076725] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:20:40 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076725/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d7d3e176e8042d9b344c0e669f27071 Command failed on smithi133 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7262f9a6-51e0-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076726] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:07:07 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076726/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076726/ sentry: https://sentry.ceph.com/organizations/ceph/?query=275f82b836274b6e8e218445959fa96e Command failed on smithi145 with status 5: 'sudo systemctl stop ceph-7fc22aae-51e0-11ed-8438-001a4aab830c@mon.smithi145' [7076727] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:10:05 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076727/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076727/ sentry: https://sentry.ceph.com/organizations/ceph/?query=12fed24cadfe47d5889c165c0f9bd5f3 Command failed on smithi046 with status 5: 'sudo systemctl stop ceph- ed7b672c-51e0-11ed-8438-001a4aab830c@mon.a' [7076728] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:09:42 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076728/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076728/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f44b9d786b1453ca835d8dcdbaaaf5e Command failed on smithi097 with status 5: 'sudo systemctl stop ceph-20924284-51e1-11ed-8438-001a4aab830c@mon.a' [7076729] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:07:51 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076729/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076729/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0bfe1c4e45c84348a56d82df8febe706 Command failed (workunit test cephadm/test_repos.sh) on smithi035 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7076730] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:06:25 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076730/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076730/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e48101efeea3455d9c3c8a46f8efc1cc Command failed on smithi040 with status 5: 'sudo systemctl stop ceph-d7327852-51e0-11ed-8438-001a4aab830c@mon.a' [7076731] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:07:43 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076731/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076731/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb5f438901d44a75aa6697b0e17e204c Command failed on smithi078 with status 5: 'sudo systemctl stop ceph- fbc7f0e8-51e0-11ed-8438-001a4aab830c@mon.a' [7076732] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:08:22 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076732/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076732/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cacf3193ad6849f4919c8ea182d0eb93 Command failed on smithi012 with status 5: 'sudo systemctl stop ceph-46687ea6-51e1-11ed-8438-001a4aab830c@mon.smithi012' [7076733] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:22:06 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076733/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076733/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0588b83d8c547289480ff4b7088986e Command failed on smithi066 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a341eb3a-51e1-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076734] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:07 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076734/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076734/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f8c3207799c40439dd88040396624ba Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5684dcd0-51e1-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076735] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:09:24 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076735/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076735/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b48a38a3f7d4f1db7da806a890c5fdd Command failed on smithi159 with status 5: 'sudo systemctl stop ceph-73be664a-51e1-11ed-8438-001a4aab830c@mon.a' [7076736] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:05:06 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076736/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076736/ sentry: https://sentry.ceph.com/organizations/ceph/?query=26e385721321441fa8bc2bd90acee649 Command failed on smithi160 with status 5: 'sudo systemctl stop ceph-5888a9b2-51e1-11ed-8438-001a4aab830c@mon.smithi160' [7076737] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:06:36 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076737/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076737/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a627e5f2f374d429d127c69419d887e Command failed on smithi017 with status 5: 'sudo systemctl stop ceph-49c9ecb0-51e1-11ed-8438-001a4aab830c@mon.a' [7076738] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:05:03 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076738/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076738/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e76078e98144b78904a0cac1e9a9bcc Command failed on smithi120 with status 5: 'sudo systemctl stop ceph-1f6ffb44-51e1-11ed-8438-001a4aab830c@mon.a' [7076739] orch/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/master} ----------------------------------------------------------------- time: 00:06:37 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076739/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076739/ sentry: https://sentry.ceph.com/organizations/ceph/?query=91928d28304e4cee972b3c2126e33b22 [Errno 2] Cannot find file on the remote 'ubuntu@smithi170.front.sepia.ceph.com': 'rook/cluster/examples/kubernetes/ceph/operator.yaml' [7076740] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:07:29 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076740/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076740/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c35a2d0653b8487aaaada014c55aaa3d Command failed on smithi074 with status 5: 'sudo systemctl stop ceph-71f10f02-51e1-11ed-8438-001a4aab830c@mon.smithi074' [7076741] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:09:41 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076741/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076741/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef21dea1faff479e9b5b415de8046b62 Command failed on smithi037 with status 5: 'sudo systemctl stop ceph-f48d9eb2-51e1-11ed-8438-001a4aab830c@mon.a' [7076742] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:11:29 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076742/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076742/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1009bea0c97244df8a1ba5d5591f86f6 Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-33d47140-51e2-11ed-8438-001a4aab830c@mon.a' [7076743] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:09:09 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076743/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076743/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1f716482c8147aebc6dc9787dc929e3 Command failed on smithi116 with status 5: 'sudo systemctl stop ceph-166be96c-51e2-11ed-8438-001a4aab830c@mon.a' [7076744] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:05:00 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076744/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076744/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9bf883a8a48946cbb98060646fef5ca3 Command failed on smithi081 with status 5: 'sudo systemctl stop ceph-8ac991e8-51e1-11ed-8438-001a4aab830c@mon.smithi081' [7076745] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:57 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076745/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076745/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42758b1a62c7449e9d5b9048b8a30f82 Command failed on smithi085 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3eb35b76-51e2-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076746] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:06:21 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076746/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076746/ sentry: https://sentry.ceph.com/organizations/ceph/?query=720444fb504e4a58abb2684145f2ad35 Command failed on smithi018 with status 5: 'sudo systemctl stop ceph- fbf4cc02-51e1-11ed-8438-001a4aab830c@mon.a' [7076747] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:05:07 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076747/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076747/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0e15a8ddecc495eb8a24a8469d6106f Command failed on smithi083 with status 5: 'sudo systemctl stop ceph-363246ce-51e2-11ed-8438-001a4aab830c@mon.a' [7076748] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:10:13 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076748/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0731ed6dfd944958d7e802d497eede0 Command failed on smithi008 with status 5: 'sudo systemctl stop ceph-b89be8d6-51e2-11ed-8438-001a4aab830c@mon.a' [7076749] orch/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:06:45 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076749/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076749/ sentry: https://sentry.ceph.com/organizations/ceph/?query=165d7838316d4ad9a780ceac890aa8bd Command failed on smithi145 with status 5: 'sudo systemctl stop ceph-4d01ecd8-51e2-11ed-8438-001a4aab830c@mon.smithi145' [7076750] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:05:21 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076750/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076750/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba5d1e623c614ce2bf0046ffecb2e6d9 Command failed on smithi071 with status 5: 'sudo systemctl stop ceph-99804c58-51e2-11ed-8438-001a4aab830c@mon.smithi071' [7076751] orch/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} ----------------------------------------------------------------- time: 00:09:38 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076751/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076751/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6682fce29d7247f0b4edf0b9db3171ab Command failed on smithi143 with status 5: 'sudo systemctl stop ceph-e602254c-51e2-11ed-8438-001a4aab830c@mon.a' [7076752] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:21:35 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076752/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076752/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1bdabdf49a4d48709c087af77baa1380 Command failed on smithi003 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f03bac18-51e2-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076753] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:21:39 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076753/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076753/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a7671725944b413d98bf07a5507121b8 Command failed on smithi099 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 808d88e6-51e2-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076756] orch/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:01 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076756/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d20ea16ac6949babe09cbd2c97498bb Command failed on smithi035 with status 5: 'sudo systemctl stop ceph-b6a63748-51e2-11ed-8438-001a4aab830c@mon.a' [7076757] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:07:01 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076757/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076757/ sentry: https://sentry.ceph.com/organizations/ceph/?query=818b8b39671045909e1ed7c83e280891 Command failed on smithi040 with status 5: 'sudo systemctl stop ceph- cc115356-51e2-11ed-8438-001a4aab830c@mon.smithi040' [7076758] orch/cephadm/smoke-singlehost/{0-distro$/{rhel_8.4_container_tools_rhel8} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:07:28 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076758/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94d9796a49e6407bb124a186640f0df9 Command failed on smithi142 with status 5: 'sudo systemctl stop ceph- cf9b53a0-51e2-11ed-8438-001a4aab830c@mon.smithi142' [7076759] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:10:07 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076759/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076759/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3541ba915f32405eac3f2b777dbc8f67 Command failed on smithi046 with status 5: 'sudo systemctl stop ceph-4bf461c6-51e3-11ed-8438-001a4aab830c@mon.a' [7076760] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} ----------------------------------------------------------------- time: 00:17:03 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076760/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076760/ sentry: https://sentry.ceph.com/organizations/ceph/?query=525d68438eed41d3a60ac01e845fc244 Command failed on smithi120 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f5ae782e-51e2-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076761] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:11:07 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076761/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076761/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eb45749f99984b49b256eefc57a55193 Command failed on smithi078 with status 5: 'sudo systemctl stop ceph-9721bf04-51e3-11ed-8438-001a4aab830c@mon.a' [7076763] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:06:45 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076763/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076763/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8c5384fdb2aa49c9bec5cbe12f119f45 Command failed on smithi017 with status 5: 'sudo systemctl stop ceph-39edf8e8-51e3-11ed-8438-001a4aab830c@mon.a' [7076764] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:08:24 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076764/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076764/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3c3bf87f5414a118e8713f9b6f94582 Command failed on smithi012 with status 5: 'sudo systemctl stop ceph-84053810-51e3-11ed-8438-001a4aab830c@mon.smithi012' [7076765] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:08:10 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076765/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076765/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c88d76d346b4d3ba6d32986c9deca96 Command failed on smithi074 with status 5: 'sudo systemctl stop ceph-5bf72e96-51e3-11ed-8438-001a4aab830c@mon.smithi074' [7076766] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:10:13 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076766/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076766/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c064912572434788a83338efc599f635 Command failed (workunit test cephadm/test_cephadm.sh) on smithi121 with status 125: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7076767] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:08:09 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076767/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076767/ sentry: https://sentry.ceph.com/organizations/ceph/?query=734e256c1d0745ce89eb90f31c32db0a Command failed on smithi160 with status 5: 'sudo systemctl stop ceph-7874aa12-51e3-11ed-8438-001a4aab830c@mon.a' [7076768] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:06:46 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076768/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076768/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d4fb31852a6488fa68a4768edab29cf Command failed on smithi081 with status 5: 'sudo systemctl stop ceph-6686832a-51e3-11ed-8438-001a4aab830c@mon.a' [7076769] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:21:44 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076769/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076769/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f699dab15a2947d5b6c8a6229fb1ef3b Command failed on smithi159 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d6f2888e-51e3-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076770] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:07:31 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076770/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076770/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b325cc0ed95140b4915b15835ed4abf6 Command failed on smithi061 with status 5: 'sudo systemctl stop ceph- ecfa3474-51e3-11ed-8438-001a4aab830c@mon.smithi061' [7076771] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:09:25 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076771/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076771/ sentry: https://sentry.ceph.com/organizations/ceph/?query=187757b24b7d42a2a8af19c93bbfa466 Command failed on smithi090 with status 5: 'sudo systemctl stop ceph-5116c558-51e4-11ed-8438-001a4aab830c@mon.a' [7076772] orch/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:06:38 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076772/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076772/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94aef478bdd74c4199d5005b65efd539 Command failed on smithi116 with status 5: 'sudo systemctl stop ceph-e3c349a4-51e3-11ed-8438-001a4aab830c@mon.a' [7076773] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:07:33 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076773/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076773/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc28000f222c40419467955b2b65332b Command failed (workunit test cephadm/test_repos.sh) on smithi037 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7076774] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:35 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076774/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076774/ sentry: https://sentry.ceph.com/organizations/ceph/?query=deb0c9fd37564fb7bbc6bf6637a56515 Command failed on smithi018 with status 5: 'sudo systemctl stop ceph- fd799916-51e3-11ed-8438-001a4aab830c@mon.a' [7076775] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:04:54 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076775/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076775/ sentry: https://sentry.ceph.com/organizations/ceph/?query=09b865690861439889fe8b39dd108a1f Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-a8371168-51e3-11ed-8438-001a4aab830c@mon.smithi043' [7076776] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:06:06 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076776/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076776/ sentry: https://sentry.ceph.com/organizations/ceph/?query=749146671d2f436b90a417d1417bcade Command failed on smithi083 with status 5: 'sudo systemctl stop ceph-0fa5cede-51e4-11ed-8438-001a4aab830c@mon.a' [7076777] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:07:32 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076777/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076777/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2fa5e2652c8b45c7ae24a4a27abdd29d Command failed on smithi027 with status 5: 'sudo systemctl stop ceph-5275090a-51e4-11ed-8438-001a4aab830c@mon.smithi027' [7076778] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:21:05 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076778/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076778/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad64dcc1304347bdbc8e55fbee826bf6 Command failed on smithi133 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 894e1ab6-51e4-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076779] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} ----------------------------------------------------------------- time: 00:22:33 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076779/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f554e63736cc43139c11223a1661e4ab Command failed on smithi036 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a788053c-51e4-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 2\'"\'"\'\'' [7076780] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:09:16 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076780/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48afb25ca55a4e37a4f0f9953e590ef1 Command failed on smithi080 with status 5: 'sudo systemctl stop ceph-d7c389a6-51e4-11ed-8438-001a4aab830c@mon.a' [7076781] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:05:09 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076781/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076781/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c4b1acac5dc45ec9bb63791a3b63ad8 Command failed on smithi035 with status 5: 'sudo systemctl stop ceph-b392f58a-51e4-11ed-8438-001a4aab830c@mon.smithi035' [7076782] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:04:59 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076782/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076782/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33c1d9dc6d3f45969d78ed211bbf7727 Command failed on smithi040 with status 5: 'sudo systemctl stop ceph-54f2e8f0-51e4-11ed-8438-001a4aab830c@mon.a' [7076783] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:06:13 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076783/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b3303ca44ec4e8daf66e7f3225f0315 Command failed on smithi045 with status 5: 'sudo systemctl stop ceph-91d20d64-51e4-11ed-8438-001a4aab830c@mon.a' [7076784] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:17:01 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076784/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076784/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a32c704873a74d0ab6bd5158be540fe6 Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad90bffa-51e4-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076785] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:06:45 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076785/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076785/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f56c19d7c1334cbca3cccbb6f669a2f0 Command failed on smithi071 with status 5: 'sudo systemctl stop ceph-975766d0-51e4-11ed-8438-001a4aab830c@mon.smithi071' [7076786] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:09:34 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076786/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076786/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc9d6f1b8703441493f9b6c4840fb181 Command failed on smithi079 with status 5: 'sudo systemctl stop ceph-0ad52732-51e5-11ed-8438-001a4aab830c@mon.a' [7076787] orch/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:06:54 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076787/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f90d9d98fa9140e88c12c8b4df5af74a Command failed on smithi102 with status 5: 'sudo systemctl stop ceph-356a70d8-51e5-11ed-8438-001a4aab830c@mon.a' [7076788] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:09:42 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076788/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076788/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b0e84c2bf204620ace9f60c3ed979e1 Command failed on smithi026 with status 5: 'sudo systemctl stop ceph-03ad81de-51e5-11ed-8438-001a4aab830c@mon.a' [7076789] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:42 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076789/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076789/ sentry: https://sentry.ceph.com/organizations/ceph/?query=92269c53c13d4390a9c978bc38e04cde Command failed on smithi143 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48834348-51e5-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076790] orch/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:07:38 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076790/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076790/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0fc98a3c8a14d4a8588acf94738fd0a Command failed on smithi097 with status 5: 'sudo systemctl stop ceph-3a6f0d50-51e5-11ed-8438-001a4aab830c@mon.smithi097' [7076791] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:05:24 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076791/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68f5e035208e4ed1b23cfcc109f3e748 Command failed on smithi057 with status 5: 'sudo systemctl stop ceph-6c3cbe72-51e5-11ed-8438-001a4aab830c@mon.a' [7076792] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:08:19 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076792/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076792/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18f92db425cb4b1386cf4e3c7a96c0d6 Command failed on smithi046 with status 5: 'sudo systemctl stop ceph-5dc154e8-51e5-11ed-8438-001a4aab830c@mon.smithi046' [7076793] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:10:30 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076793/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=99f5a335a785493aa8c740f3aac94e3b Command failed on smithi066 with status 5: 'sudo systemctl stop ceph-f3ad8a94-51e5-11ed-8438-001a4aab830c@mon.a' [7076794] orch/rook/smoke/{0-distro/ubuntu_18.04 0-kubeadm 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.6.2} ----------------------------------------------------------------- time: 00:06:35 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076794/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076794/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0be2b26334b34e089a03274d3b7f227f Command failed on smithi017 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml' [7076795] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:06:25 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076795/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076795/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43a687369bc94edbbd988b6ded628db1 Command failed on smithi078 with status 5: 'sudo systemctl stop ceph-5cc3537a-51e5-11ed-8438-001a4aab830c@mon.a' [7076796] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:07:59 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076796/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076796/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59a0b7d933384319a0ad67738c69ca3f Command failed on smithi012 with status 5: 'sudo systemctl stop ceph-b15c2e3e-51e5-11ed-8438-001a4aab830c@mon.smithi012' [7076797] orch/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} ----------------------------------------------------------------- time: 00:05:40 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076797/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076797/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a05cebe2d0854906bea864b61e652a9d Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a [7076798] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:21:39 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076798/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076798/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70e3ae2aee9d4b979b4fcb2cd94d13f1 Command failed on smithi074 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eedb12f2-51e5-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076799] orch/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:07:05 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076799/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076799/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8725906c1c945ae8527446377e9ee38 Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-823deb38-51e5-11ed-8438-001a4aab830c@mon.a' [7076800] orch/cephadm/smoke-singlehost/{0-distro$/{rhel_8.4_container_tools_3.0} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:07:56 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076800/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076800/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d168c8a8e86d4d25a063b21d26e12111 Command failed on smithi121 with status 5: 'sudo systemctl stop ceph-a77d0226-51e5-11ed-8438-001a4aab830c@mon.smithi121' [7076801] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:09:43 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076801/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076801/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a5350d28fa74100a30341d09eaf86b9 Command failed on smithi085 with status 5: 'sudo systemctl stop ceph-14ceea60-51e6-11ed-8438-001a4aab830c@mon.a' [7076802] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} ----------------------------------------------------------------- time: 00:19:16 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076802/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076802/ sentry: https://sentry.ceph.com/organizations/ceph/?query=edb02c39cde1413885fc24446e0c5267 Command failed on smithi061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ed7ea2-51e5-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076803] orch/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:09:58 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076803/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076803/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba5996f62cc04c6c9665284239409f4e Command failed on smithi018 with status 5: 'sudo systemctl stop ceph-30d030ca-51e6-11ed-8438-001a4aab830c@mon.a' [7076805] orch/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:05:20 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076805/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076805/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1485c68cbd4142c98dc6ecb311fe6138 Command failed on smithi116 with status 5: 'sudo systemctl stop ceph-929de154-51e5-11ed-8438-001a4aab830c@mon.smithi116' [7076806] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:06:35 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076806/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64f25925e668490f8c92dbc072082771 Command failed on smithi083 with status 5: 'sudo systemctl stop ceph-0693af4e-51e6-11ed-8438-001a4aab830c@mon.a' [7076807] orch/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:05:19 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076807/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076807/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af66abbbfd4043b2a38994f6a65f428e Command failed on smithi037 with status 5: 'sudo systemctl stop ceph-d45e8058-51e5-11ed-8438-001a4aab830c@mon.smithi037' [7076808] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:10:16 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076808/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076808/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8c86f3a0df94c979ea2a639f1e198b3 Command failed (workunit test cephadm/test_cephadm.sh) on smithi145 with status 125: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7076809] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_3.0 fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:08:18 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076809/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076809/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0f65facc93a4568b05bf20eadca2ace Command failed on smithi163 with status 5: 'sudo systemctl stop ceph-625d55f0-51e6-11ed-8438-001a4aab830c@mon.a' [7076810] orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:05:05 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076810/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b769aaf2b0f481ba07763e67eb182ef Command failed on smithi027 with status 5: 'sudo systemctl stop ceph-939e342c-51e6-11ed-8438-001a4aab830c@mon.smithi027' [7076811] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:21:15 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076811/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076811/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c51e4715166e4c12a36cd6ae70790375 Command failed on smithi120 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8993e300-51e6-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076812] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:06:39 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076812/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076812/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e2ff4b4013241a1b17e550485753fef Command failed on smithi090 with status 5: 'sudo systemctl stop ceph-6a479faa-51e6-11ed-8438-001a4aab830c@mon.a' [7076813] orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:20:32 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076813/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076813/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a9bf84f7142b4380b028da595d995610 Command failed on smithi071 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 635c1b26-51e6-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076814] orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:06:51 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076814/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076814/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a704968ebf0b4d0da9391dab75667771 Command failed on smithi130 with status 5: 'sudo systemctl stop ceph-8989ea3a-51e6-11ed-8438-001a4aab830c@mon.smithi130' [7076815] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:10:04 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076815/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076815/ sentry: https://sentry.ceph.com/organizations/ceph/?query=656d0bec929242dca85ea6b471ff3e8a Command failed on smithi040 with status 5: 'sudo systemctl stop ceph- eec7b256-51e6-11ed-8438-001a4aab830c@mon.a' [7076816] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:11:17 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076816/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076816/ sentry: https://sentry.ceph.com/organizations/ceph/?query=942dd7d07c234e92835009fe48a290b4 Command failed on smithi099 with status 5: 'sudo systemctl stop ceph-298e4a58-51e7-11ed-8438-001a4aab830c@mon.a' [7076817] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:07:28 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076817/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076817/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef8e69f3b5b54833b0c09b69b99169c6 Command failed (workunit test cephadm/test_repos.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7076818] orch/cephadm/smoke/{0-nvme-loop distro/rhel_8.4_container_tools_rhel8 fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:07:49 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076818/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076818/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50649d76d59948f18e5f11c1c556c85f Command failed on smithi035 with status 5: 'sudo systemctl stop ceph- ab334f50-51e6-11ed-8438-001a4aab830c@mon.a' [7076819] orch/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:05:27 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076819/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=efa9e2621d814478a6962bf16b6eb4d0 Command failed on smithi003 with status 5: 'sudo systemctl stop ceph-20baa8ae-51e7-11ed-8438-001a4aab830c@mon.smithi003' [7076820] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:08:03 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076820/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076820/ sentry: https://sentry.ceph.com/organizations/ceph/?query=378e018a54374b3283a9196918e829d3 Command failed on smithi079 with status 5: 'sudo systemctl stop ceph-21c8ebd4-51e7-11ed-8438-001a4aab830c@mon.smithi079' [7076821] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:22:33 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076821/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076821/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aee37575fb324b63a061704b01c184f4 Command failed on smithi017 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73dd70e8-51e7-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7076822] orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} ----------------------------------------------------------------- time: 00:16:45 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076822/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076822/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45023668ff3743ec8c38e0ab1e15597c Command failed on smithi102 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 359459f0-51e7-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)"\'' [7076823] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:09:29 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076823/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076823/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed90b35bed9643c9aa8dfa252b45c915 Command failed on smithi103 with status 5: 'sudo systemctl stop ceph- afb2e814-51e7-11ed-8438-001a4aab830c@mon.a' [7076824] orch/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:06:52 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076824/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e2f5082555f436792d4ed6d180ad0f0 Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-456a5852-51e7-11ed-8438-001a4aab830c@mon.a' [7076825] orch/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:05:15 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076825/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076825/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f220b05ce43741d198cfd222f144fa9e Command failed on smithi078 with status 5: 'sudo systemctl stop ceph-127af2f8-51e7-11ed-8438-001a4aab830c@mon.a' [7076826] orch/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:07:50 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076826/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076826/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bbdc86c3677476d8a32ddf362eef7e5 Command failed on smithi057 with status 5: 'sudo systemctl stop ceph-66fade4c-51e7-11ed-8438-001a4aab830c@mon.smithi057' [7076827] orch/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:10:02 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076827/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076827/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eb5fc6f02d894b7db778f2df9218c313 Command failed on smithi046 with status 5: 'sudo systemctl stop ceph-c35c8320-51e7-11ed-8438-001a4aab830c@mon.a' [7076828] orch/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:10:58 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076828/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076828/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8333daacb5a542eb94096d15e23b0478 Command failed on smithi116 with status 5: 'sudo systemctl stop ceph-e53f968a-51e7-11ed-8438-001a4aab830c@mon.a' Pass ================================================================= [7076667] orch/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_3.0} 2-node-mgr orchestrator_cli} time: 00:16:55 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076667/ [7076668] orch/cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{rhel_8} workloads/ceph_iscsi} time: 00:29:39 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076668/ [7076675] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_adoption} time: 00:11:09 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076675/ [7076717] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_adoption} time: 00:11:13 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076717/ [7076754] orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04} 2-node-mgr orchestrator_cli} time: 00:13:26 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076754/ [7076755] orch/cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{rhel_8} workloads/ceph_iscsi} time: 00:30:21 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076755/ [7076762] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_adoption} time: 00:11:20 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076762/ [7076804] orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_adoption} time: 00:11:24 info: http://pulpito.ceph.com/yuriw-2022-10-20_14:53:23-orch-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7076804/