2022-05-19T17:49:17.332 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01
2022-05-19T17:49:17.335 INFO:teuthology.results:subset: '1/2'
2022-05-19T17:49:17.335 INFO:teuthology.results:seed: '4890'
2022-05-19T18:07:47.499 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01
2022-05-19T18:07:47.521 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-05-19T18:30:49.418 INFO:teuthology.results:Tests finished! gathering results...
2022-05-19T18:30:50.179 INFO:scrape:Found 20 jobs
2022-05-19T18:30:52.253 INFO:scrape:Found 1 distinct failure reasons
2022-05-19T18:30:52.254 INFO:scrape:Failure: Command failed on smithi022 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ca40df2-d79d-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\''
2022-05-19T18:30:52.255 INFO:scrape:20 jobs: ['6841424', '6841427', '6841430', '6841441', '6841433', '6841426', '6841432', '6841425', '6841428', '6841439', '6841442', '6841443', '6841440', '6841429', '6841435', '6841437', '6841434', '6841436', '6841431', '6841438']
2022-05-19T18:30:52.256 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', 'rados:cephadm:osds/{0-distro/ubuntu_18.04']
2022-05-19T18:30:52.256 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', 'rados:cephadm:osds/{0-distro/ubuntu_18.04']
2022-05-19T18:30:52.257 INFO:scrape:
2022-05-19T18:30:52.258 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/
failed:  20
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6841424]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:18
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841424/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841424/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0247d450895b44d996996783d32eec3b

    Command failed on smithi022 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5ca40df2-d79d-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841425]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:05
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841425/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841425/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dc8b94aac1f34760a4c14bcb6893ab0d

    Command failed on smithi150 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    7bc933b0-d79d-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841426]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:10
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841426/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841426/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3981f88ecfcf486ea82012709c5e9cdf

    Command failed on smithi137 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    8d8d55f4-d79d-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841427]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:22
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841427/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841427/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e5ceb0a32a94332bb3e3862c31ef712

    Command failed on smithi033 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    cdf6f7da-d79d-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841428]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:25
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841428/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841428/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7f765a51be2c4856acd7355e4c1a61f8

    Command failed on smithi152 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    fda3eefc-d79d-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841429]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:15
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841429/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841429/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c48027f2687f44b8a2b637f8a5f5bfa1

    Command failed on smithi131 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    dae8e94e-d79d-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841430]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:13
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841430/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841430/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0fa1e956e4424e2195a2d14b1c9810a8

    Command failed on smithi007 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e547fc04-d79d-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841431]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:41
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841431/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841431/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e3f18a9f5a348eeab6e73157c4407ae

    Command failed on smithi057 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    13bda778-d79e-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841432]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:45
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841432/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841432/
sentry: https://sentry.ceph.com/organizations/ceph/?query=af73ab2b165c4f35ac71664927587a76

    Command failed on smithi071 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    530f8536-d79e-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841433]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:34
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841433/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841433/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1d381db8761143149c31ef82c68da7b2

    Command failed on smithi117 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    56cdc322-d79e-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841434]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:54
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841434/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841434/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b595fbc80b8f4ad692390ca00c4febcc

    Command failed on smithi002 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    84573a8a-d79e-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841435]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:16
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841435/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841435/
sentry: https://sentry.ceph.com/organizations/ceph/?query=29d8bf632950494bab1248d453b89f37

    Command failed on smithi100 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    a2d94d40-d79e-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841436]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:07
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841436/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841436/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cd4637960a91473a8ab1b1dda631f05e

    Command failed on smithi154 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    a5109000-d79e-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841437]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:16
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841437/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841437/
sentry: https://sentry.ceph.com/organizations/ceph/?query=90578ef8acd94c2b87824e0ad3e93590

    Command failed on smithi018 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e315747e-d79e-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841438]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:13:55
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841438/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841438/
sentry: https://sentry.ceph.com/organizations/ceph/?query=81c038caa11c4772b931bebbb170cadb

    Command failed on smithi061 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    27a7957c-d79f-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841439]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:02
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841439/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841439/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d5c4c944a4949ee9ae4855f38eb924c

    Command failed on smithi037 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3d55f954-d79f-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841440]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:13:58
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841440/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841440/
sentry: https://sentry.ceph.com/organizations/ceph/?query=650ca7294bb34dcbb0cf2affbc57c469

    Command failed on smithi116 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    618c07be-d79f-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841441]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:38
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841441/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841441/
sentry: https://sentry.ceph.com/organizations/ceph/?query=86adf795f11a4461afe38c9e7b109c6e

    Command failed on smithi039 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    6bff526e-d79f-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841442]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:41
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841442/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841442/
sentry: https://sentry.ceph.com/organizations/ceph/?query=839d264171e045aaafd87bbdf94775ad

    Command failed on smithi053 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    ef509916-d79f-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''

[6841443]  rados:cephadm:osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:16:10
info:   http://pulpito.ceph.com/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841443/
log:    http://qa-proxy.ceph.com/teuthology/adking-2022-05-19_17:46:29-rados:cephadm:osds-wip-adk2-testing-2022-05-18-1605-pacific-distro-basic-smithi/6841443/
sentry: https://sentry.ceph.com/organizations/ceph/?query=39fe6396bdda4b46bb035d597b553046

    Command failed on smithi029 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:ae8b154755bb3929b55bd3807342addf73f416ec shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    06680382-d7a0-11ec-be40-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch
    ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk
    \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk
    \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid
    $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do
    sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon
    add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep
    5 ; done\n\''