2023-12-10T05:57:56.772 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-10T05:57:56.773 INFO:teuthology.results:subset: '1/14' 2023-12-10T05:57:56.773 INFO:teuthology.results:seed: '788' 2023-12-10T05:59:28.306 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-10T05:59:28.310 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-12-10T07:19:33.779 INFO:teuthology.results:Tests finished! gathering results... 2023-12-10T07:19:35.249 INFO:scrape:Found 61 jobs 2023-12-10T07:19:37.563 INFO:scrape:Found 7 distinct failure reasons 2023-12-10T07:19:37.564 INFO:scrape:Failure: Command failed on smithi141 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74c6e664-9724-11ee-95a3-87774f69a715 --force' 2023-12-10T07:19:37.564 INFO:scrape:27 jobs: ['7485720', '7485751', '7485773', '7485740', '7485738', '7485762', '7485755', '7485750', '7485769', '7485730', '7485752', '7485741', '7485721', '7485726', '7485766', '7485775', '7485780', '7485756', '7485774', '7485732', '7485727', '7485745', '7485778', '7485760', '7485763', '7485744', '7485725'] 2023-12-10T07:19:37.564 INFO:scrape:suites intersection: [] 2023-12-10T07:19:37.565 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-wait}', '2-services/basic', '2-services/iscsi', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress2', '2-services/nfs2', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v2only', 'orch:cephadm/osds/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc}', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/with-work/{0-distro/centos_9.stream', 'orch:cephadm/with-work/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'root}', 'start', 'start}', 'task/test_ca_signed_key}', 'task/test_iscsi_container/{centos_9.stream', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_set_mon_crush_locations}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'test_iscsi_container}}'] 2023-12-10T07:19:37.565 INFO:scrape: 2023-12-10T07:19:37.565 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=479115eb9bf3e8193466e644bc7bebe1146f4825 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2023-12-10T07:19:37.565 INFO:scrape:2 jobs: ['7485777', '7485743'] 2023-12-10T07:19:37.565 INFO:scrape:suites intersection: ['agent/on', 'task/test_cephadm_repos}'] 2023-12-10T07:19:37.565 INFO:scrape:suites union: ['agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_cephadm_repos}'] 2023-12-10T07:19:37.565 INFO:scrape: 2023-12-10T07:19:37.565 INFO:scrape:Failure: Command failed on smithi002 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:479115eb9bf3e8193466e644bc7bebe1146f4825 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b8e7086-9725-11ee-95a3-87774f69a715 -- ceph rgw realm bootstrap -i -' 2023-12-10T07:19:37.565 INFO:scrape:1 jobs: ['7485729'] 2023-12-10T07:19:37.566 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'task/test_rgw_multisite}'] 2023-12-10T07:19:37.566 INFO:scrape: 2023-12-10T07:19:37.566 INFO:scrape:Failure: Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2fd6f976-9725-11ee-95a3-87774f69a715 -e sha1=479115eb9bf3e8193466e644bc7bebe1146f4825 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-12-10T07:19:37.566 INFO:scrape:6 jobs: ['7485767', '7485736', '7485771', '7485728', '7485747', '7485722'] 2023-12-10T07:19:37.566 INFO:scrape:suites intersection: [] 2023-12-10T07:19:37.566 INFO:scrape:suites union: ['1-bootstrap/17.2.0', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '2-repo_digest/repo_digest', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-12-10T07:19:37.566 INFO:scrape: 2023-12-10T07:19:37.566 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=479115eb9bf3e8193466e644bc7bebe1146f4825 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2023-12-10T07:19:37.566 INFO:scrape:1 jobs: ['7485734'] 2023-12-10T07:19:37.567 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_adoption}'] 2023-12-10T07:19:37.567 INFO:scrape: 2023-12-10T07:19:37.567 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi107 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=479115eb9bf3e8193466e644bc7bebe1146f4825 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2023-12-10T07:19:37.567 INFO:scrape:2 jobs: ['7485779', '7485758'] 2023-12-10T07:19:37.567 INFO:scrape:suites intersection: ['1-ranks/2', '1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-12-10T07:19:37.567 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-12-10T07:19:37.567 INFO:scrape: 2023-12-10T07:19:37.567 INFO:scrape:Failure: Failed to reconnect to smithi149 2023-12-10T07:19:37.567 INFO:scrape:1 jobs: ['7485742'] 2023-12-10T07:19:37.567 INFO:scrape:suites: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream'] 2023-12-10T07:19:37.568 INFO:scrape: 2023-12-10T07:19:37.569 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/ failed: 40 dead: 0 running: 0 waiting: 0 queued: 0 passed: 21 Fail ================================================================= [7485720] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:12:22 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485720/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485720/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b11fe7e58e644a497f1f52d25512805 Command failed on smithi141 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74c6e664-9724-11ee-95a3-87774f69a715 --force' [7485721] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:09:32 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485721/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485721/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de917d9bb8904f019e72539b3749734a Command failed on smithi176 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ae4b8300-9723-11ee-95a3-87774f69a715 --force' [7485722] orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:32:07 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485722/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485722/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68b46a9057da4f0e8a5aeb6aff867fe8 Command failed on smithi070 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 691162f4-9724-11ee-95a3-87774f69a715 -e sha1=479115eb9bf3e8193466e644bc7bebe1146f4825 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7485725] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:06:53 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485725/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98c393782bd246929f93c728b62af44c Command failed on smithi091 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4bdf07aa-9723-11ee-95a3-87774f69a715 --force' [7485726] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:13:10 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485726/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485726/ sentry: https://sentry.ceph.com/organizations/ceph/?query=028e3981e7da4819abbab8ddd20a68ed Command failed on smithi023 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ec500274-9724-11ee-95a3-87774f69a715 --force' [7485727] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:07:51 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485727/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485727/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88266954ada945f786dde5fd10f643eb Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7955700c-9723-11ee-95a3-87774f69a715 --force' [7485728] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:14 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485728/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485728/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d79642c20122462099971c157a7975f3 Command failed on smithi155 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1dad5dd0-9725-11ee-95a3-87774f69a715 -e sha1=479115eb9bf3e8193466e644bc7bebe1146f4825 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7485729] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:22:45 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485729/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485729/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4074f6ccc62641f7ba3824df694b2501 Command failed on smithi002 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:479115eb9bf3e8193466e644bc7bebe1146f4825 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b8e7086-9725-11ee-95a3-87774f69a715 -- ceph rgw realm bootstrap -i -' [7485730] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:16:14 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485730/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485730/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f1c3baf6ebb4dc495a9895c6fd85d90 Command failed on smithi066 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 422ccd1c-9725-11ee-95a3-87774f69a715 --force' [7485732] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:11:00 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485732/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485732/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5b6133f93ff4f9ab37ae1fcc76a6310 Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 28319c54-9724-11ee-95a3-87774f69a715 --force' [7485734] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:17:40 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485734/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485734/ sentry: https://sentry.ceph.com/organizations/ceph/?query=122ad4d9b39347b18b7bdcdc3c2c8e16 Command failed (workunit test cephadm/test_adoption.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=479115eb9bf3e8193466e644bc7bebe1146f4825 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [7485736] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:00:34 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485736/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485736/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d59b05ebf8e457bb71c6dbf50d80f3e Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75aaed6e-9724-11ee-95a3-87774f69a715 -e sha1=479115eb9bf3e8193466e644bc7bebe1146f4825 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' [7485738] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:13:36 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485738/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485738/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04af1d6fb4694a0f8ffba04435ada3b3 Command failed on smithi042 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid de03ea3c-9724-11ee-95a3-87774f69a715 --force' [7485740] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:13:21 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485740/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485740/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5b7dfcdb80743178cf2810888dc5df4 Command failed on smithi104 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c353934a-9724-11ee-95a3-87774f69a715 --force' [7485741] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485741/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485741/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0d73b1fdaa54c41bd14afbd41e07bfd Command failed on smithi149 with status 1: 'sudo yum install -y kernel' [7485742] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485742/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485742/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2619229c9d6c48c9ae05a374de32ed78 Failed to reconnect to smithi149 [7485743] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:10:33 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485743/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485743/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd0d46c9d1424ea5b25985b48c18d719 Command failed (workunit test cephadm/test_repos.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=479115eb9bf3e8193466e644bc7bebe1146f4825 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7485744] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:13:30 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485744/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485744/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfda5374e6ce4fa3acbd87e0fa103638 Command failed on smithi053 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c92aa966-9724-11ee-95a3-87774f69a715 --force' [7485745] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:12:31 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485745/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485745/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eb76254ce30749ffab617d8453ab8b37 Command failed on smithi026 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 50750b6a-9724-11ee-95a3-87774f69a715 --force' [7485747] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:34:22 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485747/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485747/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b2354cd55596428385a41aaa3682a394 Command failed on smithi017 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c1a7794-9725-11ee-95a3-87774f69a715 -e sha1=479115eb9bf3e8193466e644bc7bebe1146f4825 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7485750] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:12:59 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485750/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485750/ sentry: https://sentry.ceph.com/organizations/ceph/?query=387ad6c43c674e0abb3bfbd05fd15b95 Command failed on smithi071 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9fd0c56e-9724-11ee-95a3-87774f69a715 --force' [7485751] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:15:35 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485751/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485751/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28aadbc141db43aebaecbadce8e061ae Command failed on smithi113 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fad27caa-9724-11ee-95a3-87774f69a715 --force' [7485752] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:15:06 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485752/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485752/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abc7e5d5081848ec81d03c688a35a290 Command failed on smithi098 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e7591ce2-9724-11ee-95a3-87774f69a715 --force' [7485755] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:13:37 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485755/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485755/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7210d3d459224d938e1cba8f77d7b471 Command failed on smithi038 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d34d4674-9724-11ee-95a3-87774f69a715 --force' [7485756] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:13:33 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485756/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=032b62de2dd64f3694b3fb9152ce61ab Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d234d41e-9724-11ee-95a3-87774f69a715 --force' [7485758] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:26:41 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485758/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cbff1f0475b74918ad466e471dbe1a87 Command failed (workunit test suites/fsstress.sh) on smithi194 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=479115eb9bf3e8193466e644bc7bebe1146f4825 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7485760] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:16:49 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485760/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485760/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e92f5a3cbc14651b5dbfe3de3919006 Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48338db8-9725-11ee-95a3-87774f69a715 --force' [7485762] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:03 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485762/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485762/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f0847eb2f8c424f858d177838aa2a6e Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 075a10be-9725-11ee-95a3-87774f69a715 --force' [7485763] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:13:12 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485763/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485763/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4dd871bcd33b4be18bda5af4a1ca746a Command failed on smithi170 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b64ef6da-9724-11ee-95a3-87774f69a715 --force' [7485766] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:12:54 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485766/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485766/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9e7b8f8fe3845d3934ec7a3f0ce078a Command failed on smithi138 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed346482-9724-11ee-95a3-87774f69a715 --force' [7485767] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:58 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485767/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485767/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cf27217fa2d1480dbcc51463eff2c2fd Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2fd6f976-9725-11ee-95a3-87774f69a715 -e sha1=479115eb9bf3e8193466e644bc7bebe1146f4825 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7485769] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:13:23 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485769/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485769/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6168199949da48c298e96f9c4a847ed8 Command failed on smithi073 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e3269cb2-9724-11ee-95a3-87774f69a715 --force' [7485771] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:30:41 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485771/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485771/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d015050b9ea94005b20ef3db4085c9c1 Command failed on smithi099 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4bec263c-9724-11ee-95a3-87774f69a715 -e sha1=479115eb9bf3e8193466e644bc7bebe1146f4825 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7485773] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:12:46 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485773/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485773/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a98499fb8ab44799f6fd34c0acf3166 Command failed on smithi084 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eeb5cd8c-9724-11ee-95a3-87774f69a715 --force' [7485774] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:16:01 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485774/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485774/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a58badbd32f4409ba50a453a28fe0e92 Command failed on smithi088 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5184cf1c-9725-11ee-95a3-87774f69a715 --force' [7485775] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:15:06 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485775/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485775/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aeaa2a34b09d4248b7b2dae1c9e29755 Command failed on smithi132 with status 1: 'sudo cephadm rm-cluster --fsid e7c92ece-9724-11ee-95a3-87774f69a715 --force' [7485777] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:11:27 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485777/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485777/ sentry: https://sentry.ceph.com/organizations/ceph/?query=168e87d8bcb34d5a846f84780c9d54b0 Command failed (workunit test cephadm/test_repos.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=479115eb9bf3e8193466e644bc7bebe1146f4825 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7485778] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:13:20 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485778/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485778/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62e7b6006d6f4e03b15bb802c6f92872 Command failed on smithi131 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dacc098a-9724-11ee-95a3-87774f69a715 --force' [7485779] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:03 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485779/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7cc44e4e61d647d699d0a92932e60826 Command failed (workunit test suites/fsstress.sh) on smithi107 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=479115eb9bf3e8193466e644bc7bebe1146f4825 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7485780] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:16:26 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485780/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a5f748b7d9d47ef88a3a643a79ae008 Command failed on smithi080 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 377efc00-9725-11ee-95a3-87774f69a715 --force' Pass ================================================================= [7485723] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli} time: 00:19:54 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485723/ [7485724] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:16:35 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485724/ [7485731] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:17:59 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485731/ [7485733] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} time: 00:36:24 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485733/ [7485735] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:20:18 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485735/ [7485737] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:23:05 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485737/ [7485739] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm} time: 00:20:12 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485739/ [7485746] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:19:19 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485746/ [7485748] orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:20:31 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485748/ [7485749] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} time: 00:23:58 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485749/ [7485753] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} time: 00:20:49 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485753/ [7485754] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:20:02 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485754/ [7485757] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli} time: 00:17:44 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485757/ [7485759] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:00 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485759/ [7485761] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:17:25 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485761/ [7485764] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:29:28 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485764/ [7485765] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:22:46 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485765/ [7485768] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:20:38 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485768/ [7485770] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:07:31 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485770/ [7485772] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_adoption} time: 00:13:58 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485772/ [7485776] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:19:45 info: https://pulpito.ceph.com/adking-2023-12-10_05:57:50-orch:cephadm-wip-adk4-centos-9-runc-2023-12-09-2102-distro-default-smithi/7485776/