2023-06-28T19:52:27.739 INFO:root:teuthology version: 0.0.1.dev134+ge25bf39 2023-06-28T19:52:27.740 INFO:teuthology.results:subset: '1/14' 2023-06-28T19:52:27.741 INFO:teuthology.results:seed: '8022' 2023-06-28T20:45:48.351 INFO:root:teuthology version: 0.0.1.dev134+ge25bf39 2023-06-28T20:45:48.398 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-06-28T21:38:07.664 INFO:teuthology.results:Tests finished! gathering results... 2023-06-28T21:38:10.305 INFO:scrape:Found 92 jobs 2023-06-28T21:38:10.307 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320806/teuthology.log 2023-06-28T21:38:19.997 INFO:scrape:Found 11 distinct failure reasons 2023-06-28T21:38:19.998 INFO:scrape:Dead: None 2023-06-28T21:38:19.999 INFO:scrape:53 jobs 2023-06-28T21:38:19.999 INFO:scrape: 2023-06-28T21:38:19.999 INFO:scrape:Failure: "/var/log/ceph/d23daaf4-15f0-11ee-9b2e-001a4aab830c/ceph-mon.a.log:2023-06-28T20:19:58.203+0000 7f404b39a700 0 log_channel(cluster) log [WRN] : Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log 2023-06-28T21:38:20.000 INFO:scrape:18 jobs: ['7320770', '7320831', '7320842', '7320848', '7320826', '7320830', '7320777', '7320776', '7320810', '7320814', '7320763', '7320786', '7320827', '7320845', '7320791', '7320772', '7320768', '7320852'] 2023-06-28T21:38:20.001 INFO:scrape:suites intersection: [] 2023-06-28T21:38:20.001 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-flag}', '2-ops/rmdir-reactivate}', '2-services/nfs-ingress2', '2-services/rgw', '2-thrash', '3-final}', '3-tasks/small-objects', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'openstack}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'root}', 'start', 'start}', 'task/test_nfs}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'workloads/cephadm_iscsi}'] 2023-06-28T21:38:20.001 INFO:scrape: 2023-06-28T21:38:20.002 INFO:scrape:Failure: "/var/log/ceph/ac9b5c5e-15f2-11ee-9b2e-001a4aab830c/ceph-mon.smithi071.log:2023-06-28T20:39:14.164+0000 7f83fd67c700 10 mon.smithi071@0(leader).log v317 logging 2023-06-28T20:39:13.191877+0000 mgr.smithi193.tkuffu (mgr.24347) 87 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin" in cluster log 2023-06-28T21:38:20.002 INFO:scrape:2 jobs: ['7320782', '7320797'] 2023-06-28T21:38:20.002 INFO:scrape:suites intersection: [] 2023-06-28T21:38:20.003 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-repo_digest/defaut', '3-inline/no', '3-upgrade-mgr-staggered', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2023-06-28T21:38:20.003 INFO:scrape: 2023-06-28T21:38:20.003 INFO:scrape:Failure: "/var/log/ceph/af541ef6-15f0-11ee-9b2e-001a4aab830c/ceph-mon.smithi084.log:2023-06-28T20:24:18.832+0000 7fb31819e700 10 mon.smithi084@0(leader).log v284 logging 2023-06-28T20:24:18.600826+0000 mgr.smithi086.exikoe (mgr.14660) 37 : cephadm [WRN] Bind address in nfs.foo.0.0.smithi084.doxfpe's ganesha conf is defaulting to empty" in cluster log 2023-06-28T21:38:20.004 INFO:scrape:3 jobs: ['7320766', '7320787', '7320825'] 2023-06-28T21:38:20.004 INFO:scrape:suites intersection: ['1-start'] 2023-06-28T21:38:20.004 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/16.2.0', '1-start', '2-nfs', '2-services/nfs', '2-services/nfs-haproxy-proto', '3-final}', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8'] 2023-06-28T21:38:20.004 INFO:scrape: 2023-06-28T21:38:20.005 INFO:scrape:Failure: "/var/log/ceph/7fc13e8e-15f6-11ee-9b2e-001a4aab830c/ceph-mon.smithi089.log:2023-06-28T21:03:05.369+0000 7f5aabec8700 0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log 2023-06-28T21:38:20.005 INFO:scrape:1 jobs: ['7320843'] 2023-06-28T21:38:20.005 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun'] 2023-06-28T21:38:20.006 INFO:scrape: 2023-06-28T21:38:20.006 INFO:scrape:Failure: [Errno None] Unable to connect to port 22 on 172.21.15.62 2023-06-28T21:38:20.006 INFO:scrape:1 jobs: ['7320824'] 2023-06-28T21:38:20.007 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-06-28T21:38:20.007 INFO:scrape: 2023-06-28T21:38:20.007 INFO:scrape:Failure: "/var/log/ceph/6c3abb78-15f2-11ee-9b2e-001a4aab830c/ceph-mon.smithi090.log:2023-06-28T20:41:00.444+0000 7fd079dcc700 10 mon.smithi090@0(leader).log v410 logging 2023-06-28T20:40:59.967812+0000 mgr.smithi090.cllntc (mgr.14203) 299 : cephadm [ERR] Failed while placing nfs.foo.0.0.smithi090.wytzyu on smithi090: grace tool failed: rados_pool_create: -1" in cluster log 2023-06-28T21:38:20.008 INFO:scrape:1 jobs: ['7320783'] 2023-06-28T21:38:20.008 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-keepalive-only', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2023-06-28T21:38:20.008 INFO:scrape: 2023-06-28T21:38:20.008 INFO:scrape:Failure: "/var/log/ceph/2b15c208-15f4-11ee-9b2e-001a4aab830c/ceph-mon.c.log:2023-06-28T20:49:30.805+0000 7f416a03f700 7 mon.c@2(peon).log v311 update_from_paxos applying incremental log 311 2023-06-28T20:49:29.799423+0000 mon.a (mon.0) 829 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log 2023-06-28T21:38:20.009 INFO:scrape:3 jobs: ['7320805', '7320764', '7320774'] 2023-06-28T21:38:20.009 INFO:scrape:suites intersection: ['fixed-2'] 2023-06-28T21:38:20.009 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', 'agent/on', 'fixed-2', 'mon_election/connectivity', 'msgr/async-v1only', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'root}', 'start}'] 2023-06-28T21:38:20.010 INFO:scrape: 2023-06-28T21:38:20.010 INFO:scrape:Failure: "/var/log/ceph/5fb6d5a6-15f4-11ee-9b2e-001a4aab830c/ceph-mon.a.log:2023-06-28T20:49:59.999+0000 7f1ebb06f700 0 log_channel(cluster) log [WRN] : Health detail: HEALTH_WARN Reduced data availability: 5 pgs inactive; Degraded data redundancy: 152/594 objects degraded (25.589%), 34 pgs degraded; 1 pool(s) do not have an application enabled" in cluster log 2023-06-28T21:38:20.010 INFO:scrape:1 jobs: ['7320788'] 2023-06-28T21:38:20.011 INFO:scrape:suites: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async-v1only', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'start', 'tasks/rados_python}'] 2023-06-28T21:38:20.011 INFO:scrape: 2023-06-28T21:38:20.011 INFO:scrape:Failure: "/var/log/ceph/13508548-15f1-11ee-9b2e-001a4aab830c/ceph-mon.smithi153.log:2023-06-28T20:25:02.972+0000 7f9417ff7700 10 mon.smithi153@0(leader).log v214 logging 2023-06-28T20:25:02.969177+0000 mgr.smithi153.hmhfwt (mgr.14227) 156 : cephadm [WRN] Bind address in nfs.foo.0.0.smithi153.hnkbgh's ganesha conf is defaulting to empty" in cluster log 2023-06-28T21:38:20.011 INFO:scrape:1 jobs: ['7320765'] 2023-06-28T21:38:20.012 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-bucket', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools'] 2023-06-28T21:38:20.012 INFO:scrape: 2023-06-28T21:38:20.012 INFO:scrape:Failure: "/var/log/ceph/7ca977f2-15f1-11ee-9b2e-001a4aab830c/ceph-mon.smithi007.log:2023-06-28T20:29:42.397+0000 7f05fde95700 10 mon.smithi007@0(leader).log v262 logging 2023-06-28T20:29:41.880142+0000 mgr.smithi007.cercsl (mgr.14229) 184 : cephadm [ERR] Failed while placing nfs.foo.0.0.smithi007.hqrqty on smithi007: grace tool failed: rados_pool_create: -1" in cluster log 2023-06-28T21:38:20.013 INFO:scrape:1 jobs: ['7320775'] 2023-06-28T21:38:20.013 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0'] 2023-06-28T21:38:20.013 INFO:scrape: 2023-06-28T21:38:20.015 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/ failed: 32 dead: 53 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7320763] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:16:47 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320763/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320763/ "/var/log/ceph/aff81128-15f0-11ee-9b2e-001a4aab830c/ceph- mon.smithi053.log:2023-06-28T20:22:04.139+0000 7f5f48a47700 0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7320764] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:19:13 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320764/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320764/ "/var/log/ceph/9ea776d0-15ef-11ee-9b2e-001a4aab830c/ceph- mon.c.log:2023-06-28T20:12:30.026+0000 7fe3303ab700 7 mon.c@2(peon).log v97 update_from_paxos applying incremental log 97 2023-06-28T20:12:29.962522+0000 mon.a (mon.0) 348 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7320765] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:17:31 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320765/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320765/ "/var/log/ceph/13508548-15f1-11ee-9b2e-001a4aab830c/ceph- mon.smithi153.log:2023-06-28T20:25:02.972+0000 7f9417ff7700 10 mon.smithi153@0(leader).log v214 logging 2023-06-28T20:25:02.969177+0000 mgr.smithi153.hmhfwt (mgr.14227) 156 : cephadm [WRN] Bind address in nfs.foo.0.0.smithi153.hnkbgh's ganesha conf is defaulting to empty" in cluster log [7320766] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:41:37 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320766/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320766/ "/var/log/ceph/af541ef6-15f0-11ee-9b2e-001a4aab830c/ceph- mon.smithi084.log:2023-06-28T20:24:18.832+0000 7fb31819e700 10 mon.smithi084@0(leader).log v284 logging 2023-06-28T20:24:18.600826+0000 mgr.smithi086.exikoe (mgr.14660) 37 : cephadm [WRN] Bind address in nfs.foo.0.0.smithi084.doxfpe's ganesha conf is defaulting to empty" in cluster log [7320768] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:43:16 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320768/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320768/ "/var/log/ceph/aad6a10a-15f0-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:19:02.047+0000 7f009d2a5700 0 log_channel(cluster) log [WRN] : Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7320770] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:14:07 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320770/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320770/ "/var/log/ceph/d23daaf4-15f0-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:19:58.203+0000 7f404b39a700 0 log_channel(cluster) log [WRN] : Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7320772] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:42:49 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320772/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320772/ "/var/log/ceph/44fcc362-15f2-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:32:07.740+0000 7f5a03275700 0 log_channel(cluster) log [WRN] : Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7320774] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 01:05:18 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320774/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320774/ "/var/log/ceph/a1b76108-15f1-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:39:05.587+0000 7f59988d8700 0 log_channel(cluster) log [WRN] : Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log [7320775] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:22:28 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320775/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320775/ "/var/log/ceph/7ca977f2-15f1-11ee-9b2e-001a4aab830c/ceph- mon.smithi007.log:2023-06-28T20:29:42.397+0000 7f05fde95700 10 mon.smithi007@0(leader).log v262 logging 2023-06-28T20:29:41.880142+0000 mgr.smithi007.cercsl (mgr.14229) 184 : cephadm [ERR] Failed while placing nfs.foo.0.0.smithi007.hqrqty on smithi007: grace tool failed: rados_pool_create: -1" in cluster log [7320776] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:38:47 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320776/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320776/ "/var/log/ceph/a1b76996-15f1-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:37:09.312+0000 7fd25e80d700 0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)" in cluster log [7320777] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:19:17 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320777/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320777/ "/var/log/ceph/87b16506-15f1-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:30:38.932+0000 7f0da044f700 0 log_channel(cluster) log [WRN] : Health check failed: no active mgr (MGR_DOWN)" in cluster log [7320782] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:07 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320782/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320782/ "/var/log/ceph/ac9b5c5e-15f2-11ee-9b2e-001a4aab830c/ceph- mon.smithi071.log:2023-06-28T20:39:14.164+0000 7f83fd67c700 10 mon.smithi071@0(leader).log v317 logging 2023-06-28T20:39:13.191877+0000 mgr.smithi193.tkuffu (mgr.24347) 87 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin" in cluster log [7320783] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:22:34 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320783/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320783/ "/var/log/ceph/6c3abb78-15f2-11ee-9b2e-001a4aab830c/ceph- mon.smithi090.log:2023-06-28T20:41:00.444+0000 7fd079dcc700 10 mon.smithi090@0(leader).log v410 logging 2023-06-28T20:40:59.967812+0000 mgr.smithi090.cllntc (mgr.14203) 299 : cephadm [ERR] Failed while placing nfs.foo.0.0.smithi090.wytzyu on smithi090: grace tool failed: rados_pool_create: -1" in cluster log [7320786] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:28:47 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320786/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320786/ "/var/log/ceph/8ab74fe2-15f4-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:47:13.813+0000 7f8fdba16700 0 log_channel(cluster) log [WRN] : Health check failed: 1/4 mons down, quorum a,e,c (MON_DOWN)" in cluster log [7320787] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:16:25 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320787/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320787/ "/var/log/ceph/a02ee6de-15f2-11ee-9b2e-001a4aab830c/ceph- mon.smithi112.log:2023-06-28T20:36:04.474+0000 7fd9c5ee6700 10 mon.smithi112@0(leader).log v204 logging 2023-06-28T20:36:03.483329+0000 mgr.smithi112.rktpwt (mgr.14227) 144 : cephadm [ERR] Failed while placing nfs.foo.0.0.smithi112.wboovg on smithi112: grace tool failed: rados_pool_create: -1" in cluster log [7320788] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:29:16 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320788/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320788/ "/var/log/ceph/5fb6d5a6-15f4-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:49:59.999+0000 7f1ebb06f700 0 log_channel(cluster) log [WRN] : Health detail: HEALTH_WARN Reduced data availability: 5 pgs inactive; Degraded data redundancy: 152/594 objects degraded (25.589%), 34 pgs degraded; 1 pool(s) do not have an application enabled" in cluster log [7320791] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:18:13 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320791/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320791/ "/var/log/ceph/9b40c098-15f2-11ee-9b2e-001a4aab830c/ceph- mon.smithi120.log:2023-06-28T20:36:48.317+0000 7fb503110700 0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7320797] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:46:15 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320797/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320797/ "/var/log/ceph/f57f837c-15f3-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:53:02.745+0000 7f6d51ba0700 10 mon.a@0(leader).log v447 logging 2023-06-28T20:53:02.066385+0000 mgr.x (mgr.24634) 65 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin" in cluster log [7320805] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:41:11 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320805/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320805/ "/var/log/ceph/2b15c208-15f4-11ee-9b2e-001a4aab830c/ceph- mon.c.log:2023-06-28T20:49:30.805+0000 7f416a03f700 7 mon.c@2(peon).log v311 update_from_paxos applying incremental log 311 2023-06-28T20:49:29.799423+0000 mon.a (mon.0) 829 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log [7320810] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:32:29 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320810/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320810/ "/var/log/ceph/646477b6-15f4-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T20:59:59.994+0000 7f47ddbff700 0 log_channel(cluster) log [WRN] : Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log [7320814] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:18:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320814/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320814/ "/var/log/ceph/7fadb212-15f4-11ee-9b2e-001a4aab830c/ceph- mon.smithi023.log:2023-06-28T20:47:46.754+0000 7fa102a93700 0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7320824] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320824/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=348054b722bd4bb0aae5ef3a292d7d73 [Errno None] Unable to connect to port 22 on 172.21.15.62 [7320825] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:19:29 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320825/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320825/ "/var/log/ceph/bd13ab02-15f4-11ee-9b2e-001a4aab830c/ceph- mon.smithi033.log:2023-06-28T20:52:01.345+0000 7f35671df700 10 mon.smithi033@0(leader).log v238 logging 2023-06-28T20:52:00.865478+0000 mgr.smithi033.jqwzvp (mgr.14221) 167 : cephadm [WRN] Bind address in nfs.happy.0.0.smithi152.fonbsg's ganesha conf is defaulting to empty" in cluster log [7320826] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:43:07 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320826/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320826/ "/var/log/ceph/7110907a-15f5-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T21:18:00.917+0000 7f556023e700 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL)" in cluster log [7320827] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:45:46 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320827/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320827/ "/var/log/ceph/a5c9dd38-15f7-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T21:08:40.817+0000 7feed9fdf700 0 log_channel(cluster) log [WRN] : Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7320830] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:14:52 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320830/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320830/ "/var/log/ceph/87154bf4-15f5-11ee-9b2e-001a4aab830c/ceph- mon.smithi092.log:2023-06-28T20:56:30.363+0000 7fb82e1a7700 0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7320831] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:22:52 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320831/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320831/ "/var/log/ceph/aecceb2a-15f5-11ee-9b2e-001a4aab830c/ceph- mon.c.log:2023-06-28T20:55:15.200+0000 7f0ac99ac700 7 mon.c@2(synchronizing).log v67 update_from_paxos applying incremental log 66 2023-06-28T20:55:13.223576+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7320842] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:28:52 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320842/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320842/ "/var/log/ceph/c402f74e-15f7-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T21:10:11.679+0000 7f9609961700 0 log_channel(cluster) log [WRN] : Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7320843] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:14:56 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320843/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320843/ "/var/log/ceph/7fc13e8e-15f6-11ee-9b2e-001a4aab830c/ceph- mon.smithi089.log:2023-06-28T21:03:05.369+0000 7f5aabec8700 0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7320845] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:26:03 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320845/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320845/ "/var/log/ceph/e87cf2a6-15f6-11ee-9b2e-001a4aab830c/ceph- mon.smithi102.log:2023-06-28T21:07:09.647+0000 7fa3fe700700 0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7320848] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:44:17 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320848/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320848/ "/var/log/ceph/faac1c4a-15f6-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T21:04:58.616+0000 7ff243d55700 0 log_channel(cluster) log [WRN] : Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7320852] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:25:11 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320852/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320852/ "/var/log/ceph/827cd27c-15f7-11ee-9b2e-001a4aab830c/ceph- mon.a.log:2023-06-28T21:07:45.872+0000 7f45deeff700 0 log_channel(cluster) log [WRN] : Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log Dead ================================================================= [7320773] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320773/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320773/ Error reimaging machines: [Errno 104] Connection reset by peer [7320778] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320778/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320778/ Error reimaging machines: [Errno 104] Connection reset by peer [7320779] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320779/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320779/ Error reimaging machines: [Errno 104] Connection reset by peer [7320780] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320780/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320780/ Error reimaging machines: [Errno 104] Connection reset by peer [7320781] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320781/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320781/ Error reimaging machines: [Errno 104] Connection reset by peer [7320784] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320784/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320784/ Error reimaging machines: [Errno 104] Connection reset by peer [7320785] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320785/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320785/ Error reimaging machines: [Errno 104] Connection reset by peer [7320789] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320789/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320789/ Error reimaging machines: [Errno 104] Connection reset by peer [7320790] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320790/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320790/ Error reimaging machines: [Errno 104] Connection reset by peer [7320792] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320792/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320792/ Error reimaging machines: [Errno 104] Connection reset by peer [7320793] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320793/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320793/ Error reimaging machines: [Errno 104] Connection reset by peer [7320794] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320794/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320794/ Error reimaging machines: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/167/task [7320795] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320795/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320795/ Error reimaging machines: [Errno 104] Connection reset by peer [7320796] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320796/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320796/ Error reimaging machines: [Errno 104] Connection reset by peer [7320798] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320798/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320798/ Error reimaging machines: [Errno 104] Connection reset by peer [7320799] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320799/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320799/ Error reimaging machines: [Errno 104] Connection reset by peer [7320800] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320800/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320800/ Error reimaging machines: [Errno 104] Connection reset by peer [7320801] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320801/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320801/ Error reimaging machines: Failed to power on smithi088 [7320802] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320802/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320802/ Error reimaging machines: [Errno 104] Connection reset by peer [7320803] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320803/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320803/ Error reimaging machines: [Errno 104] Connection reset by peer [7320804] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320804/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320804/ Error reimaging machines: 'ssh_keyscan smithi062.front.sepia.ceph.com' reached maximum tries (6) after waiting for 5 seconds [7320806] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320806/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320806/ Error reimaging machines: [Errno 104] Connection reset by peer [7320807] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320807/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320807/ Error reimaging machines: [Errno 104] Connection reset by peer [7320808] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320808/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320808/ Error reimaging machines: [Errno 104] Connection reset by peer [7320809] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320809/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320809/ Error reimaging machines: [Errno 104] Connection reset by peer [7320813] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/on orchestrator_cli} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320813/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320813/ Error reimaging machines: [Errno 104] Connection reset by peer [7320815] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320815/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320815/ Error reimaging machines: [Errno 104] Connection reset by peer [7320816] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320816/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320816/ Error reimaging machines: [Errno 104] Connection reset by peer [7320817] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320817/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320817/ Error reimaging machines: [Errno 104] Connection reset by peer [7320818] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320818/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320818/ Error reimaging machines: [Errno 104] Connection reset by peer [7320819] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320819/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320819/ Error reimaging machines: [Errno 104] Connection reset by peer [7320820] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320820/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320820/ Error reimaging machines: [Errno 104] Connection reset by peer [7320821] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320821/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320821/ Error reimaging machines: [Errno 104] Connection reset by peer [7320822] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320822/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320822/ Error reimaging machines: Failed to power on smithi171 [7320823] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320823/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320823/ Error reimaging machines: [Errno 104] Connection reset by peer [7320828] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320828/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320828/ Error reimaging machines: [Errno 104] Connection reset by peer [7320829] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320829/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320829/ Error reimaging machines: [Errno 104] Connection reset by peer [7320832] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320832/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320832/ Error reimaging machines: [Errno 104] Connection reset by peer [7320833] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320833/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320833/ Error reimaging machines: [Errno 104] Connection reset by peer [7320834] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320834/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320834/ Error reimaging machines: [Errno 104] Connection reset by peer [7320835] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320835/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320835/ Error reimaging machines: [Errno 104] Connection reset by peer [7320836] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320836/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320836/ Error reimaging machines: [Errno 104] Connection reset by peer [7320837] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320837/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320837/ Error reimaging machines: [Errno 104] Connection reset by peer [7320838] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320838/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320838/ Error reimaging machines: [Errno 104] Connection reset by peer [7320839] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320839/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320839/ Error reimaging machines: [Errno 104] Connection reset by peer [7320840] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320840/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320840/ Error reimaging machines: [Errno 104] Connection reset by peer [7320841] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320841/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320841/ Error reimaging machines: [Errno 104] Connection reset by peer [7320846] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320846/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320846/ Error reimaging machines: Failed to power on smithi088 [7320847] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320847/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320847/ Error reimaging machines: SSH connection to smithi062 was lost: "while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done" [7320849] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320849/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320849/ Error reimaging machines: [Errno 104] Connection reset by peer [7320850] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320850/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320850/ Error reimaging machines: [Errno 104] Connection reset by peer [7320851] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320851/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320851/ Error reimaging machines: [7320853] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320853/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320853/ Error reimaging machines: [Errno 104] Connection reset by peer Pass ================================================================= [7320762] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:20:13 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320762/ [7320767] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/off orchestrator_cli} time: 00:22:14 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320767/ [7320769] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/basic 3-final} time: 00:17:23 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320769/ [7320771] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:32:40 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320771/ [7320811] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm} time: 00:19:11 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320811/ [7320812] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:15:31 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320812/ [7320844] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:17:59 info: http://pulpito.ceph.com/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320844/