2023-09-13T15:12:23.831 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-13T15:12:23.832 INFO:teuthology.results:subset: '1/14' 2023-09-13T15:12:23.833 INFO:teuthology.results:seed: '3193' 2023-09-13T16:54:15.204 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-13T16:54:15.230 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-14T04:35:11.436 INFO:teuthology.results:Tests finished! gathering results... 2023-09-14T04:35:18.599 INFO:scrape:Found 96 jobs 2023-09-14T04:35:20.894 INFO:scrape:Found 5 distinct failure reasons 2023-09-14T04:35:20.895 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2023-09-14T04:35:20.895 INFO:scrape:2 jobs: ['7396396', '7396328'] 2023-09-14T04:35:20.896 INFO:scrape:suites intersection: ['task/test_nfs}'] 2023-09-14T04:35:20.896 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_nfs}'] 2023-09-14T04:35:20.897 INFO:scrape: 2023-09-14T04:35:20.897 INFO:scrape:Failure: '2023-09-13T17:12:56.514641+00:00 smithi059 kernel: rcu: INFO: rcu_sched self-detected stall on CPU ' in syslog 2023-09-14T04:35:20.898 INFO:scrape:1 jobs: ['7396385'] 2023-09-14T04:35:20.898 INFO:scrape:suites: ['1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-09-14T04:35:20.898 INFO:scrape: 2023-09-14T04:35:20.899 INFO:scrape:Dead: b'2023-09-14T04:02:59.468 DEBUG:teuthology.exit:Finished running handlers' 2023-09-14T04:35:20.899 INFO:scrape:2 jobs: ['7396349', '7396379'] 2023-09-14T04:35:20.900 INFO:scrape:suites intersection: ['1-ranks/2', '1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-09-14T04:35:20.900 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-09-14T04:35:20.901 INFO:scrape: 2023-09-14T04:35:20.901 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-09-14T04:35:20.901 INFO:scrape:1 jobs: ['7396372'] 2023-09-14T04:35:20.902 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools'] 2023-09-14T04:35:20.902 INFO:scrape: 2023-09-14T04:35:20.903 INFO:scrape:Failure: Found coredumps on ubuntu@smithi141.front.sepia.ceph.com 2023-09-14T04:35:20.903 INFO:scrape:1 jobs: ['7396398'] 2023-09-14T04:35:20.903 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress2', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun'] 2023-09-14T04:35:20.904 INFO:scrape: 2023-09-14T04:35:20.908 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/ failed: 5 dead: 2 running: 0 waiting: 0 queued: 0 passed: 89 Fail ================================================================= [7396328] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:19:14 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396328/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cae7a884d2624e66af73f5cb57614e40 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7396372] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:28:42 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396372/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1455900012d44339eb843bde4b1843d reached maximum tries (301) after waiting for 300 seconds [7396385] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:40:45 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396385/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396385/ '2023-09-13T17:12:56.514641+00:00 smithi059 kernel: rcu: INFO: rcu_sched self-detected stall on CPU ' in syslog [7396396] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:19:27 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396396/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396396/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f930ca688f2b49bcb2df76ab68037b6f Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [7396398] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:15:52 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396398/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396398/ Found coredumps on ubuntu@smithi141.front.sepia.ceph.com Dead ================================================================= [7396349] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396349/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396349/ hit max job timeout [7396379] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396379/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396379/ hit max job timeout Pass ================================================================= [7396314] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:18:15 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396314/ [7396315] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:19:44 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396315/ [7396316] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:08 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396316/ [7396317] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:19:14 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396317/ [7396318] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:38:56 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396318/ [7396319] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/off orchestrator_cli} time: 00:18:02 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396319/ [7396320] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} time: 00:37:49 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396320/ [7396321] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} time: 00:17:18 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396321/ [7396322] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:13:49 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396322/ [7396323] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:18:41 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396323/ [7396324] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:25:43 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396324/ [7396325] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:46 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396325/ [7396326] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:22:54 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396326/ [7396327] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:06:12 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396327/ [7396329] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} time: 00:38:07 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396329/ [7396330] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:15:59 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396330/ [7396331] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} time: 00:35:31 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396331/ [7396332] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:18:34 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396332/ [7396333] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:17:37 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396333/ [7396334] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:14:40 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396334/ [7396335] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_orch_cli} time: 00:18:33 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396335/ [7396336] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:30:12 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396336/ [7396337] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:30:57 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396337/ [7396338] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:38:26 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396338/ [7396339] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:18:57 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396339/ [7396340] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:37:01 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396340/ [7396341] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:16:55 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396341/ [7396342] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:23:33 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396342/ [7396343] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:13:26 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396343/ [7396344] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:17:01 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396344/ [7396345] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:23 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396345/ [7396346] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:19:12 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396346/ [7396347] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_rgw_multisite} time: 00:17:22 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396347/ [7396348] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:37:27 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396348/ [7396350] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:17:21 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396350/ [7396351] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:45:43 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396351/ [7396352] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:29:59 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396352/ [7396353] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:39:04 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396353/ [7396354] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:20:11 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396354/ [7396355] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:16:46 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396355/ [7396356] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:16:14 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396356/ [7396357] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:12 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396357/ [7396358] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:18:37 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396358/ [7396359] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:42:05 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396359/ [7396360] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:40:58 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396360/ [7396361] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_adoption} time: 00:16:45 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396361/ [7396362] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:17:15 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396362/ [7396363] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:28:36 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396363/ [7396364] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:18:50 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396364/ [7396365] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:31:30 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396365/ [7396366] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:19:07 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396366/ [7396367] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/on orchestrator_cli} time: 00:20:24 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396367/ [7396368] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:18:48 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396368/ [7396369] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:13:51 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396369/ [7396370] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 00:49:22 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396370/ [7396371] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm} time: 00:18:29 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396371/ [7396373] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:36:03 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396373/ [7396374] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:11 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396374/ [7396375] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:50 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396375/ [7396376] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:27:08 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396376/ [7396377] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:14:11 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396377/ [7396378] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:09:26 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396378/ [7396380] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:19:58 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396380/ [7396381] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:43:27 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396381/ [7396382] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:36:55 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396382/ [7396383] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_extra_daemon_features} time: 00:17:46 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396383/ [7396384] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:18:54 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396384/ [7396386] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:00 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396386/ [7396387] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:20:05 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396387/ [7396388] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:21:32 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396388/ [7396389] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:27:08 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396389/ [7396390] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:30 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396390/ [7396391] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:30:18 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396391/ [7396392] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:17:47 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396392/ [7396393] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:12:25 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396393/ [7396394] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:37:23 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396394/ [7396395] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:28:12 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396395/ [7396397] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:30:32 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396397/ [7396399] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:11 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396399/ [7396400] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:18:29 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396400/ [7396401] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:19:21 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396401/ [7396402] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_orch_cli} time: 00:19:57 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396402/ [7396403] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:37:14 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396403/ [7396404] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:43:25 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396404/ [7396405] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:18:43 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396405/ [7396406] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:48 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396406/ [7396407] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli_mon} time: 00:26:25 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396407/ [7396408] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:18:34 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396408/ [7396409] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:25:00 info: http://pulpito.ceph.com/adking-2023-09-13_13:18:52-orch:cephadm-main-distro-default-smithi/7396409/