2022-07-14T12:12:48.088 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-14T12:12:48.090 INFO:teuthology.results:subset: '1/7' 2022-07-14T12:12:48.090 INFO:teuthology.results:seed: '8536' 2022-07-15T03:47:33.540 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-15T03:47:33.577 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-15T04:33:37.549 INFO:teuthology.results:Tests finished! gathering results... 2022-07-15T04:33:39.910 INFO:scrape:Found 25 jobs 2022-07-15T04:33:41.089 INFO:scrape:Found 3 distinct failure reasons 2022-07-15T04:33:41.090 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e1149808fd5503e51b84e26d74cf26215be065dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2022-07-15T04:33:41.091 INFO:scrape:1 jobs: ['6927487'] 2022-07-15T04:33:41.091 INFO:scrape:suites: ['mon_election/connectivity', 'orch:cephadm/workunits/{agent/off', 'task/test_cephadm}'] 2022-07-15T04:33:41.092 INFO:scrape: 2022-07-15T04:33:41.092 INFO:scrape:Failure: Test failure: test_update_export (tasks.cephfs.test_nfs.TestNFS) 2022-07-15T04:33:41.093 INFO:scrape:1 jobs: ['6927494'] 2022-07-15T04:33:41.093 INFO:scrape:suites: ['mon_election/classic', 'orch:cephadm/workunits/{agent/on', 'task/test_nfs}'] 2022-07-15T04:33:41.093 INFO:scrape: 2022-07-15T04:33:41.094 INFO:scrape:Failure: Command failed on smithi150 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 610d866c-0370-11ed-842e-001a4aab830c -- ceph mon dump -f json' 2022-07-15T04:33:41.094 INFO:scrape:1 jobs: ['6927481'] 2022-07-15T04:33:41.095 INFO:scrape:suites: ['1-bootstrap/16.2.4', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2022-07-15T04:33:41.095 INFO:scrape: 2022-07-15T04:33:41.097 INFO:teuthology.results:Sending results to jmulligan@redhat.com: Test Run: phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 22 Fail ================================================================= [6927481] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:09:38 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927481/ log: http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927481/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc2599b12e41457c804b0b09cf0073f1 Command failed on smithi150 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 610d866c-0370-11ed-842e-001a4aab830c -- ceph mon dump -f json' [6927487] orch:cephadm/workunits/{agent/off mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:13:38 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927487/ log: http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927487/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b339b976266042d5aabaa6efce91a6da Command failed (workunit test cephadm/test_cephadm.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e1149808fd5503e51b84e26d74cf26215be065dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [6927494] orch:cephadm/workunits/{agent/on mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:30:55 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927494/ log: http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927494/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dda10f0a379b4e8894b11059257a336d Test failure: test_update_export (tasks.cephfs.test_nfs.TestNFS) Pass ================================================================= [6927482] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:28 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927482/ [6927483] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:56:45 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927483/ [6927484] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:42 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927484/ [6927485] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:36:53 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927485/ [6927486] orch:cephadm/workunits/{agent/on mon_election/classic task/test_adoption} time: 00:09:34 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927486/ [6927488] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:29 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927488/ [6927489] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:47:30 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927489/ [6927490] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:33:41 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927490/ [6927491] orch:cephadm/workunits/{agent/on mon_election/classic task/test_cephadm_repos} time: 00:06:10 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927491/ [6927492] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:03 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927492/ [6927493] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:36:36 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927493/ [6927495] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:51 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927495/ [6927496] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:56:38 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927496/ [6927497] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:38:05 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927497/ [6927498] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:17 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927498/ [6927499] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:47:31 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927499/ [6927500] orch:cephadm/workunits/{agent/on mon_election/connectivity task/test_adoption} time: 00:09:52 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927500/ [6927501] orch:cephadm/workunits/{agent/off mon_election/classic task/test_cephadm} time: 00:19:29 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927501/ [6927502] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:18:19 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927502/ [6927503] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:40:42 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927503/ [6927504] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:36 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927504/ [6927505] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} time: 00:36:46 info: http://pulpito.ceph.com/phlogistonjohn-2022-07-12_16:56:07-orch:cephadm-wip-phlogistonjohn-testing-2022-07-12-1005-distro-default-smithi/6927505/