2023-04-04T05:19:17.434 INFO:root:teuthology version: 0.0.1.dev93+gd7c4062
2023-04-04T05:19:17.435 INFO:teuthology.results:subset: '1/13'
2023-04-04T05:19:17.435 INFO:teuthology.results:seed: '8979'
2023-04-04T05:19:36.852 INFO:root:teuthology version: 0.0.1.dev93+gd7c4062
2023-04-04T05:19:36.853 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-04T07:32:46.042 INFO:teuthology.results:Tests finished! gathering results...
2023-04-04T07:32:46.968 INFO:scrape:Found 18 jobs
2023-04-04T07:32:47.395 INFO:scrape:Found 3 distinct failure reasons
2023-04-04T07:32:47.396 INFO:scrape:Failure: Command failed on smithi027 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:27e1ef4e404073fe46b21aac3f1ae821adacafcc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3e6599b4-d2ab-11ed-9aff-001a4aab830c -- bash -c \'set -ex\n# since we don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in order to replace the fake hostnames "host.a/b/c" with\n# the actual names cephadm knows the host by within the mon spec\nceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'{datacenter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' > crush_locs\nceph orch ls --service-name mon --export > mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3 && read fakename <&4; do\n  MONSPEC="${MONSPEC//$fakename/$realname}"\ndone 3<realnames 4<fakenames\necho "$MONSPEC" > mon.yaml\ncat mon.yaml\n# now the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep 90\nceph mon dump\nceph mon dump --format json\n# verify all the crush locations got set from "ceph mon dump" output\nwhile read hostname <&3 && read crushloc <&4; do\n  ceph mon dump --format json | jq --arg hostname "$hostname" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] | select(.name == $hostname) | .crush_location == $crushloc\'"\'"\'\ndone 3<realnames 4<crush_locs\n\''
2023-04-04T07:32:47.397 INFO:scrape:2 jobs: ['7230595', '7230605']
2023-04-04T07:32:47.397 INFO:scrape:suites intersection: ['agent/on', 'mon_election/connectivity', 'task/test_set_mon_crush_locations}']
2023-04-04T07:32:47.397 INFO:scrape:suites union: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_set_mon_crush_locations}']
2023-04-04T07:32:47.397 INFO:scrape:
2023-04-04T07:32:47.398 INFO:scrape:Failure: Command failed on smithi033 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.33:/fake /mnt/foo'"
2023-04-04T07:32:47.398 INFO:scrape:2 jobs: ['7230591', '7230602']
2023-04-04T07:32:47.398 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '2-services/nfs-keepalive-only', '3-final}']
2023-04-04T07:32:47.399 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-keepalive-only', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8']
2023-04-04T07:32:47.399 INFO:scrape:
2023-04-04T07:32:47.399 INFO:scrape:Failure: Command failed on smithi118 with status 2: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce2249fe-d2aa-11ed-9aff-001a4aab830c -e sha1=27e1ef4e404073fe46b21aac3f1ae821adacafcc -- bash -c \'ceph orch ps --format json | jq --arg TARGET_ID "$TARGET_ID" -e \'"\'"\'.[] | select(.daemon_type=="mgr") | select(.container_image_id==$TARGET_ID)\'"\'"\' | jq -s > out.json\''
2023-04-04T07:32:47.400 INFO:scrape:2 jobs: ['7230604', '7230594']
2023-04-04T07:32:47.400 INFO:scrape:suites intersection: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on']
2023-04-04T07:32:47.400 INFO:scrape:suites union: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04']
2023-04-04T07:32:47.401 INFO:scrape:
2023-04-04T07:32:47.402 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/
failed:  6
dead:    0
running: 0
waiting: 0
queued:  0
passed:  12


Fail
=================================================================
[7230591]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:18:11
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230591/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230591/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a2634277ad144df7a34e9eea0421e63e

    Command failed on smithi033 with status 32: "sudo
    TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.33:/fake
    /mnt/foo'"

[7230594]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:37:09
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230594/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230594/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cb7dc2f22bcd4dbf8eea2205ac8a710d

    Command failed on smithi047 with status 2: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    c8a50dae-d2af-11ed-9aff-001a4aab830c -e
    sha1=27e1ef4e404073fe46b21aac3f1ae821adacafcc -- bash -c \'ceph orch ps
    --format json | jq --arg TARGET_ID "$TARGET_ID" -e \'"\'"\'.[] |
    select(.daemon_type=="mgr") |
    select(.container_image_id==$TARGET_ID)\'"\'"\' | jq -s > out.json\''

[7230595]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_set_mon_crush_locations}
-----------------------------------------------------------------
time:   00:20:38
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230595/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230595/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd55f5028e4f4341b4162bc5aab31840

    Command failed on smithi027 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:27e1ef4e404073fe46b21aac3f1ae821adacafcc shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3e6599b4-d2ab-11ed-9aff-001a4aab830c -- bash -c \'set -ex\n# since we
    don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in
    order to replace the fake hostnames "host.a/b/c" with\n# the actual names
    cephadm knows the host by within the mon spec\nceph orch host ls --format
    json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho
    $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'{datac
    enter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' >
    crush_locs\nceph orch ls --service-name mon --export >
    mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3
    && read fakename <&4; do\n  MONSPEC="${MONSPEC//$fakename/$realname}"\ndone
    3<realnames 4<fakenames\necho "$MONSPEC" > mon.yaml\ncat mon.yaml\n# now
    the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph
    orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls
    --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep
    90\nceph mon dump\nceph mon dump --format json\n# verify all the crush
    locations got set from "ceph mon dump" output\nwhile read hostname <&3 &&
    read crushloc <&4; do\n  ceph mon dump --format json | jq --arg hostname
    "$hostname" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] |
    select(.name == $hostname) | .crush_location == $crushloc\'"\'"\'\ndone
    3<realnames 4<crush_locs\n\''

[7230602]  orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:19:14
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230602/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230602/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd82f3dd030e4abb817d1bb303b54bd2

    Command failed on smithi077 with status 32: "sudo
    TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.77:/fake
    /mnt/foo'"

[7230604]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:23:14
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230604/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230604/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b6d6a66b3ba94150a3d4aaed977d9b95

    Command failed on smithi118 with status 2: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    ce2249fe-d2aa-11ed-9aff-001a4aab830c -e
    sha1=27e1ef4e404073fe46b21aac3f1ae821adacafcc -- bash -c \'ceph orch ps
    --format json | jq --arg TARGET_ID "$TARGET_ID" -e \'"\'"\'.[] |
    select(.daemon_type=="mgr") |
    select(.container_image_id==$TARGET_ID)\'"\'"\' | jq -s > out.json\''

[7230605]  orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_set_mon_crush_locations}
-----------------------------------------------------------------
time:   00:21:22
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230605/
log:    http://qa-proxy.ceph.com/teuthology/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230605/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bce875d5b75749f0be0e1363ddc9d61f

    Command failed on smithi039 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:27e1ef4e404073fe46b21aac3f1ae821adacafcc shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    29cb6ae2-d2ab-11ed-9aff-001a4aab830c -- bash -c \'set -ex\n# since we
    don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in
    order to replace the fake hostnames "host.a/b/c" with\n# the actual names
    cephadm knows the host by within the mon spec\nceph orch host ls --format
    json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho
    $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'{datac
    enter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' >
    crush_locs\nceph orch ls --service-name mon --export >
    mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3
    && read fakename <&4; do\n  MONSPEC="${MONSPEC//$fakename/$realname}"\ndone
    3<realnames 4<fakenames\necho "$MONSPEC" > mon.yaml\ncat mon.yaml\n# now
    the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph
    orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls
    --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep
    90\nceph mon dump\nceph mon dump --format json\n# verify all the crush
    locations got set from "ceph mon dump" output\nwhile read hostname <&3 &&
    read crushloc <&4; do\n  ceph mon dump --format json | jq --arg hostname
    "$hostname" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] |
    select(.name == $hostname) | .crush_location == $crushloc\'"\'"\'\ndone
    3<realnames 4<crush_locs\n\''



Pass
=================================================================
[7230589] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
time:   01:02:25
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230589/

[7230590] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:16:11
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230590/

[7230592] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python}
time:   01:28:27
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230592/

[7230593] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
time:   01:38:53
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230593/

[7230596] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:56:09
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230596/

[7230597] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:31:05
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230597/

[7230598] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
time:   00:42:05
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230598/

[7230599] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests}
time:   01:43:13
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230599/

[7230600] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:30:34
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230600/

[7230601] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:55:02
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230601/

[7230603] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli_mon}
time:   01:31:02
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230603/

[7230606] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   01:33:41
info:   http://pulpito.ceph.com/adking-2023-04-04_05:19:12-orch:cephadm-wip-adk-testing-2023-04-03-1541-distro-default-smithi/7230606/