2023-11-08T22:36:26.067 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T22:36:26.068 INFO:teuthology.results:subset: '1/14' 2023-11-08T22:36:26.069 INFO:teuthology.results:seed: '4894' 2023-11-08T23:49:35.848 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T23:49:35.881 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-09T00:37:39.490 INFO:teuthology.results:Tests finished! gathering results... 2023-11-09T00:37:45.556 INFO:scrape:Found 95 jobs 2023-11-09T00:37:46.519 INFO:scrape:Found 3 distinct failure reasons 2023-11-09T00:37:46.520 INFO:scrape:Failure: Command failed on smithi027 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c5dca1b39843ab5db10818164a43fe9d7a0543db shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 165dcd8c-7e91-11ee-8dbb-212e2dc638e7 -- ceph rgw realm bootstrap -i -' 2023-11-09T00:37:46.520 INFO:scrape:2 jobs: ['7452169', '7452134'] 2023-11-09T00:37:46.520 INFO:scrape:suites intersection: ['orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools'] 2023-11-09T00:37:46.521 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_extra_daemon_features}', 'task/test_rgw_multisite}'] 2023-11-09T00:37:46.521 INFO:scrape: 2023-11-09T00:37:46.522 INFO:scrape:Failure: Command failed on smithi052 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ab246710-7e90-11ee-8dbb-212e2dc638e7 -e sha1=c5dca1b39843ab5db10818164a43fe9d7a0543db -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' 2023-11-09T00:37:46.522 INFO:scrape:1 jobs: ['7452172'] 2023-11-09T00:37:46.522 INFO:scrape:suites: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools'] 2023-11-09T00:37:46.523 INFO:scrape: 2023-11-09T00:37:46.523 INFO:scrape:Failure: Command failed on smithi104 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7718266-7e8e-11ee-8dbb-212e2dc638e7 -e sha1=c5dca1b39843ab5db10818164a43fe9d7a0543db -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-11-09T00:37:46.524 INFO:scrape:1 jobs: ['7452138'] 2023-11-09T00:37:46.524 INFO:scrape:suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-11-09T00:37:46.524 INFO:scrape: 2023-11-09T00:37:46.528 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 91 Fail ================================================================= [7452134] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:15:47 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452134/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452134/ sentry: https://sentry.ceph.com/organizations/ceph/?query=984c3062a1ed4fa39da5a244ca94cf65 Command failed on smithi091 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:c5dca1b39843ab5db10818164a43fe9d7a0543db shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 533aad62-7e8f-11ee-8dbb-212e2dc638e7 -- ceph orch ls -f json' [7452138] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:31:23 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452138/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452138/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba9f88b409b34a0faa0fa6a8a425080d Command failed on smithi104 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7718266-7e8e-11ee-8dbb-212e2dc638e7 -e sha1=c5dca1b39843ab5db10818164a43fe9d7a0543db -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7452169] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:17:11 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452169/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452169/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d54870bcf3b34bb89a3def4fa13f5409 Command failed on smithi027 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:c5dca1b39843ab5db10818164a43fe9d7a0543db shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 165dcd8c-7e91-11ee-8dbb-212e2dc638e7 -- ceph rgw realm bootstrap -i -' [7452172] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:41:28 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452172/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452172/ sentry: https://sentry.ceph.com/organizations/ceph/?query=44eaf4eea56f46878f7ea196f1dde3a3 Command failed on smithi052 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ab246710-7e90-11ee-8dbb-212e2dc638e7 -e sha1=c5dca1b39843ab5db10818164a43fe9d7a0543db -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\'' Pass ================================================================= [7452135] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:21:02 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452135/ [7452136] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:25 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452136/ [7452137] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:20:26 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452137/ [7452139] orch:cephadm/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} time: 00:24:23 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452139/ [7452140] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/off orchestrator_cli} time: 00:18:42 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452140/ [7452141] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 00:42:53 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452141/ [7452142] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} time: 00:18:18 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452142/ [7452143] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:10 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452143/ [7452144] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:19:48 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452144/ [7452145] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:27:14 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452145/ [7452146] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_host_drain} time: 00:19:09 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452146/ [7452147] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:00 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452147/ [7452148] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:22:52 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452148/ [7452149] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:12:13 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452149/ [7452150] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:32 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452150/ [7452151] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} time: 00:38:13 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452151/ [7452152] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:17:11 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452152/ [7452153] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} time: 00:42:46 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452153/ [7452154] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:19:02 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452154/ [7452155] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:19:54 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452155/ [7452156] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:14:46 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452156/ [7452157] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_orch_cli} time: 00:20:42 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452157/ [7452158] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:33:39 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452158/ [7452159] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:38:28 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452159/ [7452160] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:20:17 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452160/ [7452161] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:39:05 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452161/ [7452162] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:49 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452162/ [7452163] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:18:09 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452163/ [7452164] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:23:47 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452164/ [7452165] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:13:30 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452165/ [7452166] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:18:55 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452166/ [7452167] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:10 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452167/ [7452168] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:19:10 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452168/ [7452170] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:38:26 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452170/ [7452171] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:16:51 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452171/ [7452173] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:30:31 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452173/ [7452174] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:20:18 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452174/ [7452175] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:15:48 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452175/ [7452176] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:05 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452176/ [7452177] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:16:13 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452177/ [7452178] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:16:37 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452178/ [7452179] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:18:47 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452179/ [7452180] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:41:19 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452180/ [7452181] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:41:48 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452181/ [7452182] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_adoption} time: 00:18:08 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452182/ [7452183] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:17:54 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452183/ [7452184] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:20:04 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452184/ [7452185] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:31:57 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452185/ [7452186] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:19:35 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452186/ [7452187] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 2-node-mgr agent/on orchestrator_cli} time: 00:23:00 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452187/ [7452188] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/rgw 3-final} time: 00:17:02 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452188/ [7452189] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:13:24 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452189/ [7452190] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 00:50:44 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452190/ [7452191] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm} time: 00:19:07 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452191/ [7452192] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:15:31 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452192/ [7452193] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:39:17 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452193/ [7452194] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:20 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452194/ [7452195] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:15:06 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452195/ [7452196] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:26:17 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452196/ [7452197] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:04 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452197/ [7452198] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:15:06 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452198/ [7452199] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:09:56 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452199/ [7452200] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:20:24 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452200/ [7452201] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:45:32 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452201/ [7452202] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:36:58 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452202/ [7452203] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_extra_daemon_features} time: 00:16:35 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452203/ [7452204] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:18:40 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452204/ [7452205] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:42 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452205/ [7452206] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:23:04 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452206/ [7452207] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:22:06 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452207/ [7452208] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:27:48 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452208/ [7452209] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_host_drain} time: 00:22:57 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452209/ [7452210] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:35 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452210/ [7452211] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:17:29 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452211/ [7452212] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:13:57 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452212/ [7452213] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:40:11 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452213/ [7452214] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:29:20 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452214/ [7452215] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:16:35 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452215/ [7452216] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:32:57 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452216/ [7452217] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:17:33 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452217/ [7452218] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:15:19 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452218/ [7452219] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:20:59 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452219/ [7452220] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:22:38 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452220/ [7452221] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_orch_cli} time: 00:19:46 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452221/ [7452222] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:38:10 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452222/ [7452223] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:43:58 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452223/ [7452224] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:19:41 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452224/ [7452225] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli_mon} time: 00:29:33 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452225/ [7452226] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:11 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452226/ [7452227] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:19:10 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452227/ [7452228] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:24:47 info: https://pulpito.ceph.com/adking-2023-11-08_22:36:18-orch:cephadm-wip-cv-revert-52429-main-distro-default-smithi/7452228/