2022-02-12T21:11:22.555 INFO:root:teuthology version: 1.1.0-7cebb3f2 2022-02-12T21:11:22.556 INFO:teuthology.results:subset: '1111/9999' 2022-02-12T21:11:22.556 INFO:teuthology.results:seed: '9451' 2022-02-12T21:35:55.786 INFO:root:teuthology version: 1.1.0-7cebb3f2 2022-02-12T21:35:55.790 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-13T04:13:21.947 INFO:teuthology.results:Tests finished! gathering results... 2022-02-13T04:13:27.861 INFO:scrape:Found 63 jobs 2022-02-13T04:13:29.920 INFO:scrape:Found 4 distinct failure reasons 2022-02-13T04:13:29.921 INFO:scrape:Failure: Command failed on smithi117 with status 5: 'sudo systemctl stop ceph-e276ffd6-8c4b-11ec-8c35-001a4aab830c@mon.a' 2022-02-13T04:13:29.922 INFO:scrape:9 jobs: ['6678748', '6678758', '6678718', '6678763', '6678742', '6678744', '6678719', '6678734', '6678769'] 2022-02-13T04:13:29.922 INFO:scrape:suites intersection: [] 2022-02-13T04:13:29.923 INFO:scrape:suites union: ['1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-bootstrap/octopus', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '2-repo_digest/defaut', '3-inline/no', '3-inline/yes', '3-start-upgrade', '3-upgrade-with-workload', '4-final}', '4-verify}', '4-verify}}', '4-wait', '5-upgrade-ls', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'mds', 'mon', 'mon_election/connectivity}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-02-13T04:13:29.923 INFO:scrape: 2022-02-13T04:13:29.923 INFO:scrape:Failure: Command failed on smithi109 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' 2022-02-13T04:13:29.924 INFO:scrape:12 jobs: ['6678733', '6678752', '6678761', '6678757', '6678751', '6678760', '6678773', '6678759', '6678732', '6678715', '6678756', '6678764'] 2022-02-13T04:13:29.924 INFO:scrape:suites intersection: [] 2022-02-13T04:13:29.925 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/client-keyring', '2-services/nfs-ingress', '2-services/rgw', '2-thrash', '3-final}', '3-start-upgrade', '3-tasks/small-objects', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'fixed-2', 'mode/root', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async-v2only', 'orch:cephadm/osds/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke/{0-distro/ubuntu_20.04', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{agent/off', 'orch:cephadm/workunits/{agent/on', 'root}', 'start', 'start}', 'task/test_nfs}', 'task/test_orch_cli}', 'tasks/rados_api_tests}'] 2022-02-13T04:13:29.926 INFO:scrape: 2022-02-13T04:13:29.926 INFO:scrape:Dead: b'2022-02-13T04:01:38.850 DEBUG:teuthology.exit:Finished running handlers' 2022-02-13T04:13:29.927 INFO:scrape:3 jobs: ['6678755', '6678717', '6678772'] 2022-02-13T04:13:29.927 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2022-02-13T04:13:29.927 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress2', '2-services/rgw-ingress', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8'] 2022-02-13T04:13:29.928 INFO:scrape: 2022-02-13T04:13:29.928 INFO:scrape:Failure: Command failed on smithi096 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:1acb1a750b382724a361f105093326ba14c9af0a pull' 2022-02-13T04:13:29.929 INFO:scrape:1 jobs: ['6678767'] 2022-02-13T04:13:29.929 INFO:scrape:suites: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'start', 'tasks/rados_python}'] 2022-02-13T04:13:29.930 INFO:scrape: 2022-02-13T04:13:29.932 INFO:teuthology.results:Sending results to mfritch@suse.com: Test Run: mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/ failed: 22 dead: 3 running: 0 waiting: 0 queued: 0 passed: 38 Fail ================================================================= [6678715] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-start-upgrade 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:03:00 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678715/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678715/ sentry: https://sentry.ceph.com/organizations/ceph/?query=10e8a48000a24161b1b680a0a72325e3 Command failed on smithi117 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678718] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:06:30 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678718/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678718/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e95d846362fb46b5b7da6e796eea7af6 Command failed on smithi101 with status 5: 'sudo systemctl stop ceph-94d0cfba-8c4a-11ec-8c35-001a4aab830c@mon.smithi101' [6678719] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:03:50 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678719/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678719/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a423eae7a4a6483d91797b05a33bd8ab Command failed on smithi073 with status 5: 'sudo systemctl stop ceph-3e2f2742-8c4a-11ec-8c35-001a4aab830c@mon.smithi073' [6678732] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:03:06 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678732/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678732/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ce5ecd0637b4c8596e35e72778cc000 Command failed on smithi172 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678733] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-start-upgrade 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:03:04 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678733/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678733/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76556c92b6604a21a4ff30390720cc12 Command failed on smithi109 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678734] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:06:28 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678734/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678734/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3b6774d4f0849a7bdb34e1d7aa61380 Command failed on smithi092 with status 5: 'sudo systemctl stop ceph-9c98d818-8c4b-11ec-8c35-001a4aab830c@mon.smithi092' [6678742] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:03:57 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678742/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678742/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d983f252465c433c8b36e0748c631cb1 Command failed on smithi016 with status 5: 'sudo systemctl stop ceph-94e902c8-8c4b-11ec-8c35-001a4aab830c@mon.smithi016' [6678744] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:06:31 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678744/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678744/ sentry: https://sentry.ceph.com/organizations/ceph/?query=529234e2fae04ba49db79d3f2254d326 Command failed on smithi071 with status 5: 'sudo systemctl stop ceph- fef78eb4-8c4b-11ec-8c35-001a4aab830c@mon.smithi071' [6678748] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-start-upgrade 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:03:44 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678748/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=031eb3b1c9d5402b98a6dedc5812497c Command failed on smithi117 with status 5: 'sudo systemctl stop ceph-e276ffd6-8c4b-11ec-8c35-001a4aab830c@mon.a' [6678751] orch:cephadm/workunits/{agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:04:42 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678751/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678751/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5136d5d2b9b54a0bb2c3828dc1d25dd3 Command failed on smithi084 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678752] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:03:10 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678752/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678752/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a67e4f7b7674b2b8d363119590c68cf Command failed on smithi073 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678756] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:04:58 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678756/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af171144281d4960ae526ded905dd082 Command failed on smithi143 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678757] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:04:56 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678757/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678757/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0acb12b0f98742e793301ea18781b3a1 Command failed on smithi065 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678758] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:06:26 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678758/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa806c86c8da4f87b9a3d3446fdfd2d3 Command failed on smithi036 with status 5: 'sudo systemctl stop ceph-9064288a-8c4c-11ec-8c35-001a4aab830c@mon.smithi036' [6678759] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:03:00 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678759/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678759/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd5d713fc5c544be9b900784fe7ab873 Command failed on smithi066 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678760] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:03:10 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678760/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678760/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc33533a2c5a4d4b9790a9e65a3bab50 Command failed on smithi101 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678761] orch:cephadm/workunits/{agent/off mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:04:47 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678761/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678761/ sentry: https://sentry.ceph.com/organizations/ceph/?query=506a8e80d9684daa9930eac2ef09f2ee Command failed on smithi052 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678763] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:03:45 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678763/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678763/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a94a409c19014e0e95b7dc56258f8c47 Command failed on smithi067 with status 5: 'sudo systemctl stop ceph-5631fd04-8c4c-11ec-8c35-001a4aab830c@mon.smithi067' [6678764] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-start-upgrade 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:03:05 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678764/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678764/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08f044972fb64f7daeb6acbd59216c43 Command failed on smithi019 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [6678767] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:06:13 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678767/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678767/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0cbc578d04448aba2aeb0e43538c6cd Command failed on smithi096 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:1acb1a750b382724a361f105093326ba14c9af0a pull' [6678769] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:07:32 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678769/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678769/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b1abf2b356d4265a6c474274cfc1187 Command failed on smithi033 with status 5: 'sudo systemctl stop ceph-884403e0-8c4d-11ec-8c35-001a4aab830c@mon.smithi033' [6678773] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:03:03 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678773/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678773/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c45dc8e2f6184d33ad0be4bef789b9c3 Command failed on smithi109 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' Dead ================================================================= [6678717] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678717/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678717/ hit max job timeout [6678755] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678755/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678755/ hit max job timeout [6678772] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678772/ log: http://qa-proxy.ceph.com/teuthology/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678772/ hit max job timeout Pass ================================================================= [6678716] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:10:29 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678716/ [6678720] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/off orchestrator_cli} time: 00:12:34 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678720/ [6678721] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} time: 00:26:05 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678721/ [6678722] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:11:09 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678722/ [6678723] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/basic 3-final} time: 00:09:11 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678723/ [6678724] orch:cephadm/workunits/{agent/off mon_election/classic task/test_adoption} time: 00:07:15 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678724/ [6678725] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:20:24 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678725/ [6678726] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} time: 00:32:51 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678726/ [6678727] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:19:22 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678727/ [6678728] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} time: 00:09:14 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678728/ [6678729] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:17:48 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678729/ [6678730] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:11:06 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678730/ [6678731] orch:cephadm/workunits/{agent/on mon_election/connectivity task/test_cephadm} time: 00:15:02 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678731/ [6678735] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:10:18 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678735/ [6678736] orch:cephadm/thrash/{0-distro/rhel_8.4_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} time: 00:49:25 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678736/ [6678737] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:46:46 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678737/ [6678738] orch:cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:17:43 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678738/ [6678739] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:10:24 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678739/ [6678740] orch:cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:48 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678740/ [6678741] orch:cephadm/workunits/{agent/off mon_election/classic task/test_cephadm_repos} time: 00:04:23 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678741/ [6678743] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:20:11 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678743/ [6678745] orch:cephadm/thrash/{0-distro/rhel_8.4_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:08:32 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678745/ [6678746] orch:cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:35:18 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678746/ [6678747] orch:cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:15 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678747/ [6678749] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:19:45 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678749/ [6678750] orch:cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:19:17 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678750/ [6678753] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/on orchestrator_cli} time: 00:12:30 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678753/ [6678754] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/rgw 3-final} time: 00:09:39 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678754/ [6678762] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:10:37 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678762/ [6678765] orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:18:18 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678765/ [6678766] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:44:06 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678766/ [6678768] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:11:08 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678768/ [6678770] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:11:10 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678770/ [6678771] orch:cephadm/workunits/{agent/on mon_election/connectivity task/test_adoption} time: 00:07:10 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678771/ [6678774] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/repave-all} time: 00:09:39 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678774/ [6678775] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:09:56 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678775/ [6678776] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:31:32 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678776/ [6678777] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:32:08 info: http://pulpito.ceph.com/mgfritch-2022-02-12_21:10:24-orch:cephadm-wip-mgfritch-testing-2022-02-12-1127-distro-default-smithi/6678777/