2024-02-09T08:05:45.098 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-09T08:05:45.098 INFO:teuthology.results:subset: '1/999' 2024-02-09T08:05:45.098 INFO:teuthology.results:seed: '2815' 2024-02-09T08:22:20.520 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-09T08:22:20.547 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-09T10:39:31.597 INFO:teuthology.results:Tests finished! gathering results... 2024-02-09T10:39:34.226 INFO:scrape:Found 30 jobs 2024-02-09T10:39:40.656 INFO:scrape:Found 7 distinct failure reasons 2024-02-09T10:39:40.657 INFO:scrape:Failure: Command failed on smithi123 with status 5: 'sudo systemctl stop ceph-038229c8-c726-11ee-95b6-87774f69a715@mon.smithi123' 2024-02-09T10:39:40.657 INFO:scrape:8 jobs: ['7553738', '7553730', '7553748', '7553741', '7553734', '7553726', '7553733', '7553745'] 2024-02-09T10:39:40.657 INFO:scrape:suites intersection: ['3-upgrade-with-workload'] 2024-02-09T10:39:40.657 INFO:scrape:suites union: ['1-bootstrap/16.2.5', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-final}', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4'] 2024-02-09T10:39:40.657 INFO:scrape: 2024-02-09T10:39:40.658 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2024-02-09T10:39:40.658 INFO:scrape:6 jobs: ['7553727', '7553737', '7553729', '7553747', '7553724', '7553731'] 2024-02-09T10:39:40.658 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2024-02-09T10:39:40.658 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/rgw-ingress', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2024-02-09T10:39:40.658 INFO:scrape: 2024-02-09T10:39:40.658 INFO:scrape:Failure: Command failed on smithi100 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb08281e-c725-11ee-95b6-87774f69a715 -e sha1=ec2ab61d2e47f3ebc86a512475c5fe8aeb013930 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2024-02-09T10:39:40.658 INFO:scrape:1 jobs: ['7553742'] 2024-02-09T10:39:40.658 INFO:scrape:suites: ['1-bootstrap/octopus', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools'] 2024-02-09T10:39:40.658 INFO:scrape: 2024-02-09T10:39:40.659 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ec2ab61d2e47f3ebc86a512475c5fe8aeb013930 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-02-09T10:39:40.659 INFO:scrape:2 jobs: ['7553725', '7553739'] 2024-02-09T10:39:40.659 INFO:scrape:suites intersection: ['orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] 2024-02-09T10:39:40.659 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] 2024-02-09T10:39:40.659 INFO:scrape: 2024-02-09T10:39:40.659 INFO:scrape:Failure: Failed to reconnect to smithi079 2024-02-09T10:39:40.659 INFO:scrape:1 jobs: ['7553728'] 2024-02-09T10:39:40.659 INFO:scrape:suites: ['1-install/nautilus-v1only', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'distro$/{ubuntu_18.04}', 'mon_election/classic', 'msgr-failures/osd-delay', 'orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}'] 2024-02-09T10:39:40.659 INFO:scrape: 2024-02-09T10:39:40.659 INFO:scrape:Failure: timeout expired in wait_until_healthy 2024-02-09T10:39:40.660 INFO:scrape:2 jobs: ['7553720', '7553744'] 2024-02-09T10:39:40.660 INFO:scrape:suites intersection: ['3-upgrade/simple', '4-wait', '5-upgrade-ls', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9'] 2024-02-09T10:39:40.660 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade/simple', '4-wait', '5-upgrade-ls', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9'] 2024-02-09T10:39:40.660 INFO:scrape: 2024-02-09T10:39:40.660 INFO:scrape:Failure: 'package_manager_version' 2024-02-09T10:39:40.660 INFO:scrape:1 jobs: ['7553723'] 2024-02-09T10:39:40.660 INFO:scrape:suites: ['cluster/{fixed-3', 'openstack}', 'orch:cephadm/rbd_iscsi/{base/install', 'pool/datapool', 'supported-random-distro$/{centos_8}', 'workloads/ceph_iscsi}'] 2024-02-09T10:39:40.660 INFO:scrape: 2024-02-09T10:39:40.662 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/ failed: 21 dead: 0 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [7553720] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} ----------------------------------------------------------------- time: 00:19:54 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553720/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553720/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d343a7408c704df3b15ccf9b16afc1b8 timeout expired in wait_until_healthy [7553723] orch:cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{centos_8} workloads/ceph_iscsi} ----------------------------------------------------------------- time: 00:12:22 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553723/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553723/ sentry: https://sentry.ceph.com/organizations/ceph/?query=838fd010159441e5bf8abc644e8b2cc6 'package_manager_version' [7553724] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:44:50 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553724/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553724/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad03d3338ba24620b175c8d6e579f551 reached maximum tries (301) after waiting for 300 seconds [7553725] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:19:28 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553725/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5431f39f4724b889e23e7143605a5e2 Command failed (workunit test cephadm/test_cephadm.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ec2ab61d2e47f3ebc86a512475c5fe8aeb013930 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7553726] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:27:39 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553726/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553726/ sentry: https://sentry.ceph.com/organizations/ceph/?query=90784a3d02a147bd934961dceb7d31fe Command failed on smithi183 with status 5: 'sudo systemctl stop ceph-e4b0832e-c724-11ee-95b6-87774f69a715@mon.smithi183' [7553727] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:40:54 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553727/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553727/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f825d2dfc9b46709c3054a6b3736cc7 reached maximum tries (301) after waiting for 300 seconds [7553728] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553728/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553728/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0676dd320e7245f3a30ebcecd8f1cf18 Failed to reconnect to smithi079 [7553729] orch:cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:45:34 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553729/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553729/ sentry: https://sentry.ceph.com/organizations/ceph/?query=251db893658f4bd09fcf86afddcc4169 reached maximum tries (301) after waiting for 300 seconds [7553730] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:26:41 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553730/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553730/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f426dd28781459e814bd379100b0fdc Command failed on smithi149 with status 5: 'sudo systemctl stop ceph-64ca936a-c725-11ee-95b6-87774f69a715@mon.smithi149' [7553731] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:48:15 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553731/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553731/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f62e4a9c41b74d0185f8b9378f79d140 reached maximum tries (301) after waiting for 300 seconds [7553733] orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:23:41 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553733/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553733/ sentry: https://sentry.ceph.com/organizations/ceph/?query=450cb6a2d0fe4870bdee65b648692a0b Command failed on smithi167 with status 5: 'sudo systemctl stop ceph- ede3f4bc-c724-11ee-95b6-87774f69a715@mon.smithi167' [7553734] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:27:26 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553734/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553734/ sentry: https://sentry.ceph.com/organizations/ceph/?query=47fc60bda66744968ef415e99fb5c3dd Command failed on smithi155 with status 5: 'sudo systemctl stop ceph-8c9c2322-c725-11ee-95b6-87774f69a715@mon.smithi155' [7553737] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:42:16 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553737/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553737/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a14af78749b24bf7b2a27e4a88a0370f reached maximum tries (301) after waiting for 300 seconds [7553738] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:26:42 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553738/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553738/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64964d35693d4bf2b15afa8879ee7041 Command failed on smithi123 with status 5: 'sudo systemctl stop ceph-038229c8-c726-11ee-95b6-87774f69a715@mon.smithi123' [7553739] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:21:10 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553739/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553739/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59e49d4020db4baf9da0d7ba00830588 Command failed (workunit test cephadm/test_cephadm.sh) on smithi028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ec2ab61d2e47f3ebc86a512475c5fe8aeb013930 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7553741] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:26:46 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553741/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553741/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7191597c7f9d4548b18b065fece391e2 Command failed on smithi039 with status 5: 'sudo systemctl stop ceph-5fb1dbda-c726-11ee-95b6-87774f69a715@mon.smithi039' [7553742] orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 01:36:00 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553742/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553742/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0310659c2ac479eafc7b076f1a6ebcf Command failed on smithi100 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb08281e-c725-11ee-95b6-87774f69a715 -e sha1=ec2ab61d2e47f3ebc86a512475c5fe8aeb013930 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7553744] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:47 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553744/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553744/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba3eff5579d649acab84f0e4c4161309 timeout expired in wait_until_healthy [7553745] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:26:14 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553745/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553745/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d037d7f8b21f4cf8bb23cdb65ec91cfc Command failed on smithi097 with status 5: 'sudo systemctl stop ceph-626a8fb6-c726-11ee-95b6-87774f69a715@mon.smithi097' [7553747] orch:cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:45:17 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553747/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553747/ sentry: https://sentry.ceph.com/organizations/ceph/?query=865182c8b0f040739962cd6cfe06f4e9 reached maximum tries (301) after waiting for 300 seconds [7553748] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:26:19 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553748/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe03ff16a51a42ba98b613694654a8d7 Command failed on smithi158 with status 5: 'sudo systemctl stop ceph-e5b560b2-c726-11ee-95b6-87774f69a715@mon.smithi158' Pass ================================================================= [7553721] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 01:26:08 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553721/ [7553722] orch:cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:32:28 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553722/ [7553732] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:55:17 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553732/ [7553735] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:37:54 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553735/ [7553736] orch:cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:22:13 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553736/ [7553740] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:54:48 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553740/ [7553743] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:49:21 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553743/ [7553746] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} time: 02:11:57 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553746/ [7553749] orch:cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:55:11 info: https://pulpito.ceph.com/gabrioux-2024-02-09_08:05:22-orch:cephadm-wip-guits-testing-2024-02-08-0728-pacific-distro-default-smithi/7553749/