2023-11-09T14:07:16.884 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T14:07:16.885 INFO:teuthology.results:subset: '1/8' 2023-11-09T14:07:16.885 INFO:teuthology.results:seed: '8457' 2023-11-09T15:45:48.175 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T15:45:48.203 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-09T16:38:52.123 INFO:teuthology.results:Tests finished! gathering results... 2023-11-09T16:38:56.879 INFO:scrape:Found 102 jobs 2023-11-09T16:38:58.896 INFO:scrape:Found 5 distinct failure reasons 2023-11-09T16:38:58.897 INFO:scrape:Failure: Command failed on smithi133 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9dd47a5fb6354a92c6d5691f81aa1e4f83f7d7b4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2c87c7d4-7f0e-11ee-8dbb-212e2dc638e7 -- ceph mon dump -f json' 2023-11-09T16:38:58.897 INFO:scrape:1 jobs: ['7452887'] 2023-11-09T16:38:58.898 INFO:scrape:suites: ['0-nvme-loop', 'agent/off', 'fixed-2', 'mon_election/classic', 'orch:cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8', 'start}'] 2023-11-09T16:38:58.898 INFO:scrape: 2023-11-09T16:38:58.898 INFO:scrape:Failure: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) 2023-11-09T16:38:58.899 INFO:scrape:3 jobs: ['7452876', '7452920', '7452960'] 2023-11-09T16:38:58.899 INFO:scrape:suites intersection: ['task/test_nfs}'] 2023-11-09T16:38:58.899 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0', 'task/test_nfs}'] 2023-11-09T16:38:58.900 INFO:scrape: 2023-11-09T16:38:58.900 INFO:scrape:Failure: timeout expired in wait_until_healthy 2023-11-09T16:38:58.900 INFO:scrape:1 jobs: ['7452964'] 2023-11-09T16:38:58.901 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-11-09T16:38:58.901 INFO:scrape: 2023-11-09T16:38:58.902 INFO:scrape:Failure: '2023-11-09T15:25:13.318674+00:00 smithi059 kernel: rcu: INFO: rcu_sched self-detected stall on CPU ' in syslog 2023-11-09T16:38:58.902 INFO:scrape:1 jobs: ['7452912'] 2023-11-09T16:38:58.902 INFO:scrape:suites: ['1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-11-09T16:38:58.903 INFO:scrape: 2023-11-09T16:38:58.903 INFO:scrape:Failure: ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 15777 (apt-get)'] 2023-11-09T16:38:58.903 INFO:scrape:1 jobs: ['7452932'] 2023-11-09T16:38:58.904 INFO:scrape:suites: ['1-start', '2-services/rgw', '3-final}', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04}'] 2023-11-09T16:38:58.904 INFO:scrape: 2023-11-09T16:38:58.907 INFO:teuthology.results:Sending results to gabrioux@ibm.com: Test Run: gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/ failed: 6 dead: 1 running: 0 waiting: 0 queued: 0 passed: 95 Fail ================================================================= [7452876] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:28:40 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452876/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452876/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7cd36b8c17f432fb6f864c96883bc34 Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) [7452887] orch:cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:13:36 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452887/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452887/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a27b4ba1ff6644c89b5aa8c86adc4dc5 Command failed on smithi133 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:9dd47a5fb6354a92c6d5691f81aa1e4f83f7d7b4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2c87c7d4-7f0e-11ee-8dbb-212e2dc638e7 -- ceph mon dump -f json' [7452912] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:33:21 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452912/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452912/ '2023-11-09T15:25:13.318674+00:00 smithi059 kernel: rcu: INFO: rcu_sched self-detected stall on CPU ' in syslog [7452920] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:25:03 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452920/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452920/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb948aef3461499b98713c58cca2b18e Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) [7452960] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:24:51 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452960/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452960/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73d982b460824e038d16b072f2222405 Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) [7452964] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:26:54 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452964/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452964/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0915d1999b9747218ffe4ad83d590872 timeout expired in wait_until_healthy Dead ================================================================= [7452932] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:03:09 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452932/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452932/ sentry: https://sentry.ceph.com/organizations/ceph/?query=90895cec74374f9884844f98ef6eb669 ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 15777 (apt-get)'] Pass ================================================================= [7452874] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:15:45 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452874/ [7452875] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:30:01 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452875/ [7452877] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:16:24 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452877/ [7452878] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:59 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452878/ [7452879] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:33:28 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452879/ [7452880] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 2-node-mgr agent/off orchestrator_cli} time: 00:19:11 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452880/ [7452881] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 00:40:47 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452881/ [7452882] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} time: 00:18:28 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452882/ [7452883] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:27:19 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452883/ [7452884] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:31:22 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452884/ [7452885] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli} time: 00:20:18 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452885/ [7452886] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:22:07 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452886/ [7452888] orch:cephadm/thrash/{0-distro/rhel_8.4_container_tools_rhel8 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} time: 00:41:53 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452888/ [7452889] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:43:15 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452889/ [7452890] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli_mon} time: 00:41:25 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452890/ [7452891] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:19:52 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452891/ [7452892] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:27:16 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452892/ [7452893] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:21:21 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452893/ [7452894] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:46 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452894/ [7452895] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:25:44 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452895/ [7452896] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:21:51 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452896/ [7452897] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:16 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452897/ [7452898] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:52:29 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452898/ [7452899] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:23:58 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452899/ [7452900] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:27:10 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452900/ [7452901] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:09:01 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452901/ [7452902] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:44:33 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452902/ [7452903] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/on mon_election/connectivity task/test_adoption} time: 00:17:05 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452903/ [7452904] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:20:49 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452904/ [7452905] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:29:01 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452905/ [7452906] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm} time: 00:21:58 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452906/ [7452907] orch:cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:20:17 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452907/ [7452908] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:39:58 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452908/ [7452909] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:07:30 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452909/ [7452910] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:17:02 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452910/ [7452911] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:53 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452911/ [7452913] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:17:15 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452913/ [7452914] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:33:41 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452914/ [7452915] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:35:21 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452915/ [7452916] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:36:49 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452916/ [7452917] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:17:53 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452917/ [7452918] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:21:20 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452918/ [7452919] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:27:35 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452919/ [7452921] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:18:15 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452921/ [7452922] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:29:27 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452922/ [7452923] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/off mon_election/classic task/test_orch_cli} time: 00:21:14 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452923/ [7452924] orch:cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:19:14 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452924/ [7452925] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:16:48 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452925/ [7452926] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:39:42 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452926/ [7452927] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:43:03 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452927/ [7452928] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:34:46 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452928/ [7452929] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:26:07 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452929/ [7452930] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:54 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452930/ [7452931] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04} 2-node-mgr agent/on orchestrator_cli} time: 00:13:58 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452931/ [7452933] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:44:19 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452933/ [7452934] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:31:09 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452934/ [7452935] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} time: 00:22:52 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452935/ [7452936] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:18:03 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452936/ [7452937] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} time: 00:29:38 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452937/ [7452938] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:22:32 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452938/ [7452939] orch:cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:17:42 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452939/ [7452940] orch:cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:21:35 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452940/ [7452941] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:15:36 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452941/ [7452942] orch:cephadm/thrash/{0-distro/rhel_8.4_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:45:41 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452942/ [7452943] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:37:13 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452943/ [7452944] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_adoption} time: 00:13:29 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452944/ [7452945] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:33:27 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452945/ [7452946] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/on mon_election/connectivity task/test_cephadm} time: 00:24:35 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452946/ [7452947] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:20:06 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452947/ [7452948] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:36 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452948/ [7452949] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:35:41 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452949/ [7452950] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:34:42 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452950/ [7452951] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:30:28 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452951/ [7452952] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm_repos} time: 00:11:54 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452952/ [7452953] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:19:33 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452953/ [7452954] orch:cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:20:49 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452954/ [7452955] orch:cephadm/thrash/{0-distro/rhel_8.4_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 00:55:03 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452955/ [7452956] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:45:40 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452956/ [7452957] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:14 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452957/ [7452958] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:20:45 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452958/ [7452959] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:24:42 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452959/ [7452961] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:20:54 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452961/ [7452962] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} time: 00:27:58 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452962/ [7452963] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_orch_cli} time: 00:17:34 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452963/ [7452965] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:53:08 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452965/ [7452966] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:18:35 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452966/ [7452967] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:27:09 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452967/ [7452968] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:44:59 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452968/ [7452969] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:41:21 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452969/ [7452970] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/off mon_election/classic task/test_orch_cli_mon} time: 00:37:20 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452970/ [7452971] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:17:21 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452971/ [7452972] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:31:32 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452972/ [7452973] orch:cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:23:29 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452973/ [7452974] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:58 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452974/ [7452975] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:37:46 info: https://pulpito.ceph.com/gabrioux-2023-11-09_14:05:38-orch:cephadm-wip-63490-quincy-distro-default-smithi/7452975/