2024-02-20T14:30:31.637 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-20T14:30:31.637 INFO:teuthology.results:subset: '1/999' 2024-02-20T14:30:31.637 INFO:teuthology.results:seed: '5185' 2024-02-20T15:50:07.449 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-20T15:50:07.485 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-20T18:08:18.813 INFO:teuthology.results:Tests finished! gathering results... 2024-02-20T18:08:20.979 INFO:scrape:Found 30 jobs 2024-02-20T18:08:30.671 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568889/teuthology.log 2024-02-20T18:08:32.738 INFO:scrape:Found 5 distinct failure reasons 2024-02-20T18:08:32.739 INFO:scrape:Failure: "2024-02-20T16:11:31.679205+0000 mon.a (mon.0) 589 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log 2024-02-20T18:08:32.739 INFO:scrape:17 jobs: ['7568912', '7568909', '7568898', '7568916', '7568893', '7568914', '7568899', '7568896', '7568906', '7568895', '7568900', '7568913', '7568905', '7568907', '7568887', '7568891', '7568902'] 2024-02-20T18:08:32.739 INFO:scrape:suites intersection: [] 2024-02-20T18:08:32.739 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-install/mimic-v1only', '1-install/octopus', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-verify}', '4-verify}}', '4-wait', '5-upgrade-ls', 'backoff/peering', 'backoff/peering_and_degraded', 'centos_8.stream_container_tools', 'ceph', 'clusters/{openstack', 'conf/{client', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/packaged', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v2only', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/ubuntu_18.04', 'orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/ubuntu_18.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados', 'roles', 'start', 'task/test_nfs}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/{0-from/v16.2.4', 'thrashers/careful', 'thrashers/default', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}', 'workloads/snaps-few-objects}'] 2024-02-20T18:08:32.739 INFO:scrape: 2024-02-20T18:08:32.739 INFO:scrape:Failure: expected string or bytes-like object 2024-02-20T18:08:32.739 INFO:scrape:6 jobs: ['7568903', '7568915', '7568892', '7568908', '7568897', '7568888'] 2024-02-20T18:08:32.739 INFO:scrape:suites intersection: ['1-start', '2-thrash', 'fixed-2', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'root}'] 2024-02-20T18:08:32.740 INFO:scrape:suites union: ['1-start', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'fixed-2', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'root}'] 2024-02-20T18:08:32.740 INFO:scrape: 2024-02-20T18:08:32.740 INFO:scrape:Dead: None 2024-02-20T18:08:32.740 INFO:scrape:1 jobs: ['7568889'] 2024-02-20T18:08:32.740 INFO:scrape: 2024-02-20T18:08:32.740 INFO:scrape:Failure: Command failed on smithi157 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a23f2fac-d009-11ee-95bb-87774f69a715 -e sha1=19854089e18d4f65dda2b6cd74e737367c2514bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2024-02-20T18:08:32.740 INFO:scrape:1 jobs: ['7568910'] 2024-02-20T18:08:32.740 INFO:scrape:suites: ['1-bootstrap/octopus', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools'] 2024-02-20T18:08:32.740 INFO:scrape: 2024-02-20T18:08:32.740 INFO:scrape:Failure: Command failed on smithi115 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.115:/fake /mnt/foo -o port=2999'" 2024-02-20T18:08:32.741 INFO:scrape:1 jobs: ['7568894'] 2024-02-20T18:08:32.741 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress2', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2024-02-20T18:08:32.741 INFO:scrape: 2024-02-20T18:08:32.742 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/ failed: 25 dead: 1 running: 0 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [7568887] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} ----------------------------------------------------------------- time: 00:33:00 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568887/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568887/ "2024-02-20T15:08:43.367373+0000 mon.a (mon.0) 761 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7568888] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:39:53 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568888/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568888/ expected string or bytes-like object [7568891] orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:32:29 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568891/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568891/ "2024-02-20T15:09:44.662682+0000 mon.smithi019 (mon.0) 329 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7568892] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:43:20 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568892/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568892/ expected string or bytes-like object [7568893] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} ----------------------------------------------------------------- time: 00:52:27 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568893/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568893/ "2024-02-20T15:50:00.000220+0000 mon.a (mon.0) 678 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 81/609 objects degraded (13.300%), 28 pgs degraded" in cluster log [7568894] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:21:57 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568894/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=749a294f04db4915a09f534390d8dff2 Command failed on smithi115 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.115:/fake /mnt/foo -o port=2999'" [7568895] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:30:50 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568895/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568895/ "2024-02-20T15:18:08.367658+0000 mon.a (mon.0) 286 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log [7568896] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:55:34 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568896/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568896/ "2024-02-20T15:40:00.000361+0000 mon.a (mon.0) 1606 : cluster [WRN] pg 1.0 is stuck peering for 7m, current state peering, last acting [11,7,6]" in cluster log [7568897] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:42:17 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568897/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568897/ expected string or bytes-like object [7568898] orch:cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:40:55 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568898/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568898/ "2024-02-20T16:10:00.000416+0000 mon.a (mon.0) 3155 : cluster [WRN] [WRN] POOL_FULL: 1 pool(s) full" in cluster log [7568899] orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:32:59 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568899/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568899/ "2024-02-20T15:47:17.194740+0000 mon.smithi038 (mon.0) 358 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7568900] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:19:01 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568900/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568900/ "2024-02-20T15:51:48.838094+0000 mon.a (mon.0) 468 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log [7568902] orch:cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:23:16 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568902/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568902/ "2024-02-20T15:59:43.249743+0000 mon.smithi027 (mon.0) 108 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7568903] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:46:22 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568903/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568903/ expected string or bytes-like object [7568905] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:33:32 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568905/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568905/ "2024-02-20T16:20:00.000128+0000 mon.smithi005 (mon.0) 575 : cluster [WRN] pg 3.14 is active+recovery_wait+undersized+degraded+remapped, acting [0,5]" in cluster log [7568906] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:33:51 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568906/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568906/ "2024-02-20T16:30:00.000104+0000 mon.a (mon.0) 1249 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 4208/50786 objects degraded (8.286%), 2 pgs degraded" in cluster log [7568907] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:25:31 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568907/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568907/ sentry: https://sentry.ceph.com/organizations/ceph/?query=857a88e7f1b44031be441dde9a50e496 "2024-02-20T16:12:18.817891+0000 mon.smithi154 (mon.0) 294 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7568908] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:37:42 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568908/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568908/ expected string or bytes-like object [7568909] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:37:51 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568909/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568909/ "2024-02-20T16:17:36.308276+0000 mon.a (mon.0) 992 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7568910] orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:19:27 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568910/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568910/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1138ab16e7754513adb76f9da8886c1d Command failed on smithi157 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a23f2fac-d009-11ee-95bb-87774f69a715 -e sha1=19854089e18d4f65dda2b6cd74e737367c2514bd -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7568912] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} ----------------------------------------------------------------- time: 00:20:18 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568912/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568912/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ea795948b9e4bfb944d2cf3a9c73732 "2024-02-20T16:11:31.679205+0000 mon.a (mon.0) 589 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7568913] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:28:39 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568913/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568913/ "2024-02-20T16:19:10.881460+0000 mon.a (mon.0) 1198 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7568914] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 02:09:35 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568914/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568914/ "2024-02-20T16:30:00.000255+0000 mon.a (mon.0) 1083 : cluster [WRN] pg 2.4 is active+recovery_wait+undersized+degraded+remapped, acting [10,2]" in cluster log [7568915] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:49:40 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568915/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568915/ expected string or bytes-like object [7568916] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:16:26 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568916/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568916/ "2024-02-20T16:13:58.995262+0000 mon.a (mon.0) 466 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log Dead ================================================================= [7568889] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568889/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568889/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7568890] orch:cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} time: 00:30:40 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568890/ [7568901] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:59 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568901/ [7568904] orch:cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} time: 00:26:26 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568904/ [7568911] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:48:20 info: https://pulpito.ceph.com/gabrioux-2024-02-20_14:26:47-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568911/