2022-10-05T12:27:00.865 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-05T12:27:00.867 INFO:teuthology.results:subset: '140/256'
2022-10-05T12:27:00.867 INFO:teuthology.results:seed: '5999'
2022-10-05T12:42:49.988 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-05T12:42:49.992 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-10-05T13:29:53.436 INFO:teuthology.results:Tests finished! gathering results...
2022-10-05T13:29:53.940 INFO:scrape:Found 10 jobs
2022-10-05T13:29:54.890 INFO:scrape:Found 1 distinct failure reasons
2022-10-05T13:29:54.891 INFO:scrape:Failure: Command failed on smithi043 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 64847016-44ad-11ed-8433-001a4aab830c -e sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"
2022-10-05T13:29:54.892 INFO:scrape:10 jobs: ['7054846', '7054850', '7054849', '7054847', '7054841', '7054842', '7054848', '7054844', '7054845', '7054843']
2022-10-05T13:29:54.892 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}']
2022-10-05T13:29:54.893 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4']
2022-10-05T13:29:54.893 INFO:scrape:
2022-10-05T13:29:54.894 INFO:teuthology.results:Sending results to ridave@redhat.com: Test Run: rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/
failed:  10
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7054841]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:17:01
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054841/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054841/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9970cdf2067f4d83ad6108317a74c4ae

    Command failed on smithi096 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    a75e77da-44ab-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

[7054842]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:35:24
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054842/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054842/
sentry: https://sentry.ceph.com/organizations/ceph/?query=19a42de1ae2e4b60bd8646f998178a1a

    Command failed on smithi003 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    1871ac30-44ac-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7054843]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:34:56
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054843/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054843/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13e48449a9534ac497c18f4f155a2814

    Command failed on smithi120 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4885057a-44ac-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7054844]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:18:16
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054844/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054844/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7cb46447b972444aaa86d61f62558344

    Command failed on smithi090 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    1a7cf9f2-44ad-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

[7054845]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:35:15
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054845/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054845/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2e52bd4111f449ef84d46ffce103c67e

    Command failed on smithi103 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    7cc24360-44ad-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7054846]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:17:18
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054846/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054846/
sentry: https://sentry.ceph.com/organizations/ceph/?query=da2dcb45f5244e94b78f1347b5155aa4

    Command failed on smithi043 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    64847016-44ad-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

[7054847]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:34:33
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054847/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054847/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5101cfec61ca426fb550ee2c43df968d

    Command failed on smithi159 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    8e3107b2-44ad-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7054848]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:17:10
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054848/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054848/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6114865720e345d0b5f211aa5998aa1f

    Command failed on smithi161 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    c97c4a2a-44ad-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

[7054849]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:35:25
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054849/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054849/
sentry: https://sentry.ceph.com/organizations/ceph/?query=29443a28cb6048bc9bbc74e5ae177dee

    Command failed on smithi027 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e5bd5f26-44ad-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7054850]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:17:15
info:   http://pulpito.ceph.com/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054850/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-10-05_12:24:04-fs-wip-rishabh-testing-30Sep2022-testing-default-smithi/7054850/
sentry: https://sentry.ceph.com/organizations/ceph/?query=958402bb5e174b0ea4a943cf429aa05e

    Command failed on smithi055 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    d5e776c2-44ad-11ed-8433-001a4aab830c -e
    sha1=bf18ff11925632de7e414b73e468dc52e4483e4e -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"