2024-02-23T00:45:08.383 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-23T00:45:08.383 INFO:teuthology.results:subset: '111/120000'
2024-02-23T00:45:08.383 INFO:teuthology.results:seed: '5210'
2024-02-23T00:58:31.411 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-23T00:58:31.416 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-23T04:23:47.491 INFO:teuthology.results:Tests finished! gathering results...
2024-02-23T04:23:47.993 INFO:scrape:Found 6 jobs
2024-02-23T04:23:48.529 INFO:scrape:Found 3 distinct failure reasons
2024-02-23T04:23:48.529 INFO:scrape:Timeout 120 running ceph --cluster ceph health --format=json
2024-02-23T04:23:48.529 INFO:scrape:2 jobs: ['7571119', '7571125']
2024-02-23T04:23:48.529 INFO:scrape:suites intersection: ['upgrade:reef-p2p/reef-p2p-parallel/{point-to-point-upgrade']
2024-02-23T04:23:48.529 INFO:scrape:suites union: ['supported-all-distro/centos_8}', 'supported-all-distro/ubuntu_latest}', 'upgrade:reef-p2p/reef-p2p-parallel/{point-to-point-upgrade']
2024-02-23T04:23:48.530 INFO:scrape:
2024-02-23T04:23:48.530 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=v18.2.1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh'
2024-02-23T04:23:48.530 INFO:scrape:3 jobs: ['7571127', '7571121', '7571123']
2024-02-23T04:23:48.530 INFO:scrape:suites intersection: ['1-ceph-install/reef', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack']
2024-02-23T04:23:48.530 INFO:scrape:suites union: ['1-ceph-install/reef', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'objectstore/bluestore-stupid', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack']
2024-02-23T04:23:48.530 INFO:scrape:
2024-02-23T04:23:48.531 INFO:scrape:Failure: Command failed on smithi100 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap'
2024-02-23T04:23:48.531 INFO:scrape:1 jobs: ['7571129']
2024-02-23T04:23:48.531 INFO:scrape:suites: ['1-ceph-install/reef', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack']
2024-02-23T04:23:48.531 INFO:scrape:
2024-02-23T04:23:48.532 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/
failed:  6
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7571119]  upgrade:reef-p2p/reef-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8}
-----------------------------------------------------------------
time:   00:57:28
info:   https://pulpito.ceph.com/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571119/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571119/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b782d854839245fe9a752d7b9caa697a

    Command failed on smithi027 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph health --format=json'

[7571121]  upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   02:54:35
info:   https://pulpito.ceph.com/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571121/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571121/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c8e9d3f05ae6490e81c56c328739bb10

    Command failed (workunit test rbd/test_librbd_python.sh) on smithi195 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=v18.2.1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/rbd/test_librbd_python.sh'

[7571123]  upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   02:59:40
info:   https://pulpito.ceph.com/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571123/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571123/
sentry: https://sentry.ceph.com/organizations/ceph/?query=250998834d1346d69f2ceff9006f8166

    Command failed (workunit test rbd/test_librbd_python.sh) on smithi079 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=v18.2.1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/rbd/test_librbd_python.sh'

[7571125]  upgrade:reef-p2p/reef-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest}
-----------------------------------------------------------------
time:   00:52:50
info:   https://pulpito.ceph.com/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571125/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571125/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7bd53e4de6f642e3b832ad77f0a0f1b4

    Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph health --format=json'

[7571127]  upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   03:14:09
info:   https://pulpito.ceph.com/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571127/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571127/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c2a98223df9f4617b7233797d0312304

    Command failed (workunit test rbd/test_librbd_python.sh) on smithi077 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=v18.2.1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/rbd/test_librbd_python.sh'

[7571129]  upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   00:07:47
info:   https://pulpito.ceph.com/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571129/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-22_21:35:10-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7571129/
sentry: https://sentry.ceph.com/organizations/ceph/?query=33f6f3e0ed2a435fb69c42a640c9aa36

    Command failed on smithi100 with status 1: 'sudo MALLOC_CHECK_=3 adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd
    --cluster ceph --mkfs --mkkey -i 0 --monmap
    /home/ubuntu/cephtest/ceph.monmap'