2024-02-24T00:57:41.566 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-24T00:57:41.567 INFO:teuthology.results:subset: '111/120000'
2024-02-24T00:57:41.567 INFO:teuthology.results:seed: '1050'
2024-02-24T01:05:11.050 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-24T01:05:11.053 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-24T01:24:12.480 INFO:teuthology.results:Tests finished! gathering results...
2024-02-24T01:24:12.909 INFO:scrape:Found 4 jobs
2024-02-24T01:24:13.015 INFO:scrape:Found 2 distinct failure reasons
2024-02-24T01:24:13.016 INFO:scrape:Failure: An attempt to upgrade from a higher version to a lower one will always fail. Hint: check tags in the target git branch.
2024-02-24T01:24:13.016 INFO:scrape:3 jobs: ['7572884', '7572885', '7572883']
2024-02-24T01:24:13.016 INFO:scrape:suites intersection: ['1-ceph-install/reef', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:reef-p2p:reef-p2p-stress-split/{0-cluster/{openstack']
2024-02-24T01:24:13.016 INFO:scrape:suites union: ['1-ceph-install/reef', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'objectstore/bluestore-stupid', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:reef-p2p:reef-p2p-stress-split/{0-cluster/{openstack']
2024-02-24T01:24:13.016 INFO:scrape:
2024-02-24T01:24:13.016 INFO:scrape:Failure: Command failed on smithi037 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap'
2024-02-24T01:24:13.016 INFO:scrape:1 jobs: ['7572886']
2024-02-24T01:24:13.016 INFO:scrape:suites: ['1-ceph-install/reef', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:reef-p2p:reef-p2p-stress-split/{0-cluster/{openstack']
2024-02-24T01:24:13.017 INFO:scrape:
2024-02-24T01:24:13.017 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7572883]  upgrade:reef-p2p:reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   00:09:51
info:   https://pulpito.ceph.com/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/7572883/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/7572883/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a35f99daee374231af75e1fd75523c49

    An attempt to upgrade from a higher version to a lower one will always
    fail. Hint: check tags in the target git branch.

[7572884]  upgrade:reef-p2p:reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   00:09:41
info:   https://pulpito.ceph.com/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/7572884/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/7572884/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d12b38f49ffa48b18112530e165ecbe7

    An attempt to upgrade from a higher version to a lower one will always
    fail. Hint: check tags in the target git branch.

[7572885]  upgrade:reef-p2p:reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   00:09:30
info:   https://pulpito.ceph.com/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/7572885/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/7572885/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2809ba7a409c49c88b78a7a7438127a6

    An attempt to upgrade from a higher version to a lower one will always
    fail. Hint: check tags in the target git branch.

[7572886]  upgrade:reef-p2p:reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   00:07:56
info:   https://pulpito.ceph.com/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/7572886/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-24_00:55:19-upgrade:reef-p2p:reef-p2p-stress-split-pacific-release-distro-default-smithi/7572886/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5699f2af66c84dcdb24c5266f5b14022

    Command failed on smithi037 with status 1: 'sudo MALLOC_CHECK_=3 adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd
    --cluster ceph --mkfs --mkkey -i 0 --monmap
    /home/ubuntu/cephtest/ceph.monmap'