2023-12-06T06:18:32.701 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-06T06:18:32.701 INFO:teuthology.results:subset: '1/14' 2023-12-06T06:18:32.702 INFO:teuthology.results:seed: '2836' 2023-12-06T07:19:20.543 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-06T07:19:20.570 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-12-06T19:19:20.583 INFO:teuthology.results:Tests finished! gathering results... 2023-12-06T19:19:28.352 INFO:scrape:Found 98 jobs 2023-12-06T19:19:33.797 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.797 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.797 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.823 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.823 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.823 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.848 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.848 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.848 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.872 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.872 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:33.873 WARNING:scrape:Saw end of BT but not start 2023-12-06T19:19:34.817 INFO:scrape:Found 6 distinct failure reasons 2023-12-06T19:19:34.818 INFO:scrape:Failure: Command failed on smithi164 with status 1: 'sudo yum -y install ceph' 2023-12-06T19:19:34.818 INFO:scrape:12 jobs: ['7479340', '7479347', '7479319', '7479316', '7479373', '7479295', '7479366', '7479336', '7479337', '7479346', '7479329', '7479290'] 2023-12-06T19:19:34.818 INFO:scrape:suites intersection: [] 2023-12-06T19:19:34.818 INFO:scrape:suites union: ['1-start', '2-node-mgr', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'orchestrator_cli}', 'root}', 'start', 'task/test_ca_signed_key}', 'task/test_cephadm}', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rotate-keys}'] 2023-12-06T19:19:34.818 INFO:scrape: 2023-12-06T19:19:34.819 INFO:scrape:Failure: Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) 2023-12-06T19:19:34.819 INFO:scrape:1 jobs: ['7479382'] 2023-12-06T19:19:34.819 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_orch_cli}'] 2023-12-06T19:19:34.819 INFO:scrape: 2023-12-06T19:19:34.819 INFO:scrape:Failure: Command failed on smithi028 with status 126: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0482646fcf78576d871bc87843257d82edb79541 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5faab3b2-9406-11ee-95a2-87774f69a715 -- ceph orch host add smithi190' 2023-12-06T19:19:34.819 INFO:scrape:6 jobs: ['7479333', '7479354', '7479367', '7479326', '7479308', '7479302'] 2023-12-06T19:19:34.819 INFO:scrape:suites intersection: [] 2023-12-06T19:19:34.819 INFO:scrape:suites union: ['0-nvme-loop', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-ops/repave-all}', '2-repo_digest/repo_digest', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade/simple', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic', 'mon_election/connectivity}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'task/test_rgw_multisite}', 'tasks/{0-from/quincy'] 2023-12-06T19:19:34.820 INFO:scrape: 2023-12-06T19:19:34.820 INFO:scrape:Failure: Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d3c9b9ca-9404-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-12-06T19:19:34.820 INFO:scrape:7 jobs: ['7479317', '7479328', '7479383', '7479332', '7479293', '7479371', '7479349'] 2023-12-06T19:19:34.820 INFO:scrape:suites intersection: [] 2023-12-06T19:19:34.820 INFO:scrape:suites union: ['1-bootstrap/17.2.0', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-12-06T19:19:34.820 INFO:scrape: 2023-12-06T19:19:34.820 INFO:scrape:Failure: Command failed on smithi177 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-pytest python3-pytest python3-pytest python3-pytest' 2023-12-06T19:19:34.820 INFO:scrape:2 jobs: ['7479365', '7479300'] 2023-12-06T19:19:34.821 INFO:scrape:suites intersection: ['fixed-2', 'start', 'tasks/rados_python}'] 2023-12-06T19:19:34.821 INFO:scrape:suites union: ['fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'start', 'tasks/rados_python}'] 2023-12-06T19:19:34.821 INFO:scrape: 2023-12-06T19:19:34.821 INFO:scrape:Dead: b'2023-12-06T19:16:52.269 DEBUG:teuthology.exit:Finished running handlers' 2023-12-06T19:19:34.821 INFO:scrape:1 jobs: ['7479372'] 2023-12-06T19:19:34.821 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2023-12-06T19:19:34.821 INFO:scrape: 2023-12-06T19:19:34.825 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/ failed: 28 dead: 1 running: 0 waiting: 0 queued: 0 passed: 69 Fail ================================================================= [7479290] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:13:29 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479290/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479290/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e68fda04c74452383fa35b81f7d6d1b Command failed on smithi193 with status 1: 'sudo yum -y install ceph' [7479293] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:30:58 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479293/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479293/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45f19eda1f0540119ba7bdff8cd6b927 Command failed on smithi116 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d2584ba4-9401-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7479295] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/off orchestrator_cli} ----------------------------------------------------------------- time: 00:14:02 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479295/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479295/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66b4a26b806a49b79f7f5c6dcb0e75bd Command failed on smithi187 with status 1: 'sudo yum -y install ceph' [7479300] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:12:12 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479300/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479300/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94885b06d53e4114b1e784c370257e85 Command failed on smithi087 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-pytest python3-pytest python3-pytest python3-pytest' [7479302] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:29:17 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479302/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479302/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8177b7c78d8d4b5aad6432d2e73d8ba7 Command failed on smithi072 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09a7347a-9403-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7479308] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:36:48 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479308/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479308/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7880af15d9414a67acbfd126f87c7911 Command failed on smithi040 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f675b9d0-9402-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7479316] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:13:14 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479316/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479316/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67e96c4eb8b347059aa735164d985dd9 Command failed on smithi151 with status 1: 'sudo yum -y install ceph' [7479317] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:30:28 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479317/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479317/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4528d14d0a5476da3d65786489c5bd2 Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d3c9b9ca-9404-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7479319] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:13:12 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479319/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479319/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5a7cf7044a10423e8e097727fc6d9b05 Command failed on smithi187 with status 1: 'sudo yum -y install ceph' [7479326] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:18:47 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479326/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479326/ sentry: https://sentry.ceph.com/organizations/ceph/?query=166b94cbca7e4df190c7d9efb4a7636f Command failed on smithi081 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:0482646fcf78576d871bc87843257d82edb79541 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b851e6c6-9405-11ee-95a2-87774f69a715 -- ceph rgw realm bootstrap -i -' [7479328] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:43:35 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479328/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcf69086f3154c9db53eec5d6ec4356d Command failed on smithi094 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7a48bdc-9405-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo'" [7479329] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:13:13 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479329/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479329/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5ebab80fe534869bb217a6696a0fcaf Command failed on smithi135 with status 1: 'sudo yum -y install ceph' [7479332] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:29:15 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479332/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479332/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8bcfaccd0ec94e39b27c92d6acd49bb5 Command failed on smithi069 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bfc3c874-9406-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7479333] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:14:11 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479333/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479333/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b958cb1f3c244aafaff29e5884a3b0f9 Command failed on smithi028 with status 126: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:0482646fcf78576d871bc87843257d82edb79541 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5faab3b2-9406-11ee-95a2-87774f69a715 -- ceph orch host add smithi190' [7479336] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:13:31 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479336/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85fab96d00fa4043b8a49e0f0ba817b1 Command failed on smithi088 with status 1: 'sudo yum -y install ceph' [7479337] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:13:15 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479337/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479337/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0de8285a91f04956ba2d51e8402d89bc Command failed on smithi136 with status 1: 'sudo yum -y install ceph' [7479340] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:13:48 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479340/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479340/ sentry: https://sentry.ceph.com/organizations/ceph/?query=901381bc80014ff4a3e15b33797347ed Command failed on smithi164 with status 1: 'sudo yum -y install ceph' [7479346] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:12:53 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479346/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479346/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42ee6fd2ab9c469db2bb4d621e4f86c0 Command failed on smithi125 with status 1: 'sudo yum -y install ceph' [7479347] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:13:11 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479347/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479347/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e392de1f24004271b9f3793bd8c4be2c Command failed on smithi191 with status 1: 'sudo yum -y install ceph' [7479349] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:34:30 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479349/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c83849533ccc412bb4793f5ddc902bdd Command failed on smithi172 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c6e2358-9407-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7479354] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:30:50 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479354/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479354/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a834b7df9dd84bc689738c86fbbe6914 Command failed on smithi002 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 54a84a5e-9408-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7479365] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:12:10 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479365/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479365/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e9cfe3ec9da46049ce65957c7c5b6d1 Command failed on smithi177 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-pytest python3-pytest python3-pytest python3-pytest' [7479366] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:13:51 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479366/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479366/ sentry: https://sentry.ceph.com/organizations/ceph/?query=795ece80393b4f9eb23214e4af5888a1 Command failed on smithi105 with status 1: 'sudo yum -y install ceph' [7479367] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:21 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479367/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479367/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87df317e8d5943a188932d4d9dbaa201 Command failed on smithi052 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48628e70-9409-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7479371] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:26:11 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479371/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479371/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b7975dbd8ea94ed2a543c1043a9cf43b Command failed on smithi155 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0c98e95c-9409-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7479373] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:13:15 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479373/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479373/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7e834019ea14c1bb83bcec78972138a Command failed on smithi190 with status 1: 'sudo yum -y install ceph' [7479382] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:21:00 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479382/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479382/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0401408148541df80eb72b837bd4704 Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) [7479383] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:27:48 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479383/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=429f1fbf2be449e0b9d432cef21a6874 Command failed on smithi113 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75857eb6-940a-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' Dead ================================================================= [7479372] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479372/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479372/ hit max job timeout Pass ================================================================= [7479289] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:22:05 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479289/ [7479291] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:10 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479291/ [7479292] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:22:48 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479292/ [7479294] orch:cephadm/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} time: 00:31:20 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479294/ [7479296] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 00:39:21 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479296/ [7479297] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/basic 3-final} time: 00:18:54 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479297/ [7479298] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:13:46 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479298/ [7479299] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:20:08 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479299/ [7479301] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_host_drain} time: 00:23:59 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479301/ [7479303] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:23:46 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479303/ [7479304] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:15:39 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479304/ [7479305] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:39 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479305/ [7479306] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} time: 00:38:13 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479306/ [7479307] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:16:33 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479307/ [7479309] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:20:38 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479309/ [7479310] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:20:30 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479310/ [7479311] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:23:02 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479311/ [7479312] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:14:53 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479312/ [7479313] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:32:39 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479313/ [7479314] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:37:04 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479314/ [7479315] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:21:40 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479315/ [7479318] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:18:12 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479318/ [7479320] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:24:43 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479320/ [7479321] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:12:55 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479321/ [7479322] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:18:41 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479322/ [7479323] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:26:17 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479323/ [7479324] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:19:14 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479324/ [7479325] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:37:33 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479325/ [7479327] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:17:31 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479327/ [7479330] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:19:07 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479330/ [7479331] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:17:09 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479331/ [7479334] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:32 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479334/ [7479335] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:20:37 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479335/ [7479338] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_adoption} time: 00:14:59 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479338/ [7479339] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:18:43 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479339/ [7479341] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:32:19 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479341/ [7479342] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:19:53 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479342/ [7479343] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/on orchestrator_cli} time: 00:17:10 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479343/ [7479344] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:18:43 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479344/ [7479345] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:14:00 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479345/ [7479348] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:14:46 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479348/ [7479350] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:06 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479350/ [7479351] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:16:11 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479351/ [7479352] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:27:33 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479352/ [7479353] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:08:09 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479353/ [7479355] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:15:16 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479355/ [7479356] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_cephadm_timeout} time: 00:19:10 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479356/ [7479357] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:22:00 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479357/ [7479358] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:43:10 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479358/ [7479359] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:37:02 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479359/ [7479360] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:20:12 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479360/ [7479361] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:17:54 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479361/ [7479362] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:32 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479362/ [7479363] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:21:47 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479363/ [7479364] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:21:57 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479364/ [7479368] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:16:24 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479368/ [7479369] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:26 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479369/ [7479370] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:37:45 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479370/ [7479374] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:17:47 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479374/ [7479375] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:30 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479375/ [7479376] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:20:25 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479376/ [7479377] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:25:41 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479377/ [7479378] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:22:21 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479378/ [7479379] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:38:02 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479379/ [7479380] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:45:22 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479380/ [7479381] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:20:31 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479381/ [7479384] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:19:59 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479384/ [7479385] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_orch_cli_mon} time: 00:25:46 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479385/ [7479386] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:23:52 info: https://pulpito.ceph.com/adking-2023-12-06_06:12:16-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7479386/