2023-12-06T14:30:14.959 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-06T14:30:14.959 INFO:teuthology.results:subset: '1/14' 2023-12-06T14:30:14.959 INFO:teuthology.results:seed: '2836' 2023-12-06T14:44:01.413 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-06T14:44:01.416 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-12-06T15:32:05.255 INFO:teuthology.results:Tests finished! gathering results... 2023-12-06T15:32:06.043 INFO:scrape:Found 13 jobs 2023-12-06T15:32:09.390 INFO:scrape:Found 3 distinct failure reasons 2023-12-06T15:32:09.390 INFO:scrape:Failure: Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c349e5c8-9447-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-12-06T15:32:09.391 INFO:scrape:10 jobs: ['7480276', '7480274', '7480271', '7480278', '7480281', '7480277', '7480270', '7480279', '7480275', '7480272'] 2023-12-06T15:32:09.391 INFO:scrape:suites intersection: [] 2023-12-06T15:32:09.391 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-12-06T15:32:09.391 INFO:scrape: 2023-12-06T15:32:09.391 INFO:scrape:Failure: Command failed on smithi176 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5'" 2023-12-06T15:32:09.391 INFO:scrape:1 jobs: ['7480269'] 2023-12-06T15:32:09.391 INFO:scrape:suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-12-06T15:32:09.392 INFO:scrape: 2023-12-06T15:32:09.392 INFO:scrape:Failure: Command failed on smithi045 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0482646fcf78576d871bc87843257d82edb79541 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5fef0cf6-9447-11ee-95a2-87774f69a715 -- ceph rgw realm bootstrap -i -' 2023-12-06T15:32:09.392 INFO:scrape:1 jobs: ['7480273'] 2023-12-06T15:32:09.392 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_rgw_multisite}'] 2023-12-06T15:32:09.392 INFO:scrape: 2023-12-06T15:32:09.393 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/ failed: 12 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7480269] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:29:49 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480269/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480269/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ee6180c0be4444badd0000919edf8be Command failed on smithi176 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5'" [7480270] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:28:28 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480270/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480270/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f005f20f61947b5bc69f0e131474550 Command failed on smithi152 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 062f0e3c-9447-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7480271] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:37:00 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480271/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480271/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b859a523012a49fa8fd6a4eb99936dd2 Command failed on smithi116 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4424379a-9446-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7480272] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:29:49 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480272/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480272/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac130c7de6ac46898f55c8ad77b5b253 Command failed on smithi078 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f2cdfbe-9447-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7480273] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:19:20 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480273/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480273/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c213dc28bf2441a797f9b584dd5d4f3a Command failed on smithi045 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:0482646fcf78576d871bc87843257d82edb79541 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5fef0cf6-9447-11ee-95a2-87774f69a715 -- ceph rgw realm bootstrap -i -' [7480274] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:44:54 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480274/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480274/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a7d2b618326743a6a41e3a3e63bf2902 Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f399ab6-9447-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.rgw | keys\'"\'"\' | grep $sha1\'' [7480275] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:29:53 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480275/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480275/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7290787ac8ca4e7aaef6d6aa431ebf3d Command failed on smithi091 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0c74f52-9447-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7480276] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:36:11 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480276/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480276/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95d36fc22e154105889972205d397753 Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c349e5c8-9447-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7480277] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:28:44 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480277/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480277/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4e2fcf8ea6b4cf6a194159cd854e75e Command failed on smithi061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bd539ac-9448-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7480278] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:29:48 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480278/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480278/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f1f2dff4d734806a8a4a1613d0ffbf2 Command failed on smithi154 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69164924-9448-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7480279] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:26:14 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480279/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480279/ sentry: https://sentry.ceph.com/organizations/ceph/?query=520e9f462b544f1e95b94046335b3c06 Command failed on smithi077 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efbd4c76-9447-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7480281] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:28:31 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480281/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480281/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e733efca1bf74343b2916e7f83db9df9 Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b74050fe-9448-11ee-95a2-87774f69a715 -e sha1=0482646fcf78576d871bc87843257d82edb79541 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' Pass ================================================================= [7480280] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli} time: 00:17:26 info: https://pulpito.ceph.com/adking-2023-12-06_14:29:19-orch:cephadm-wip-adk-testing-2023-12-05-1905-distro-default-smithi/7480280/