2024-02-02T02:15:37.463 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-02T02:15:37.463 INFO:teuthology.results:subset: '111/120000'
2024-02-02T02:15:37.463 INFO:teuthology.results:seed: '5741'
2024-02-02T02:27:31.475 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-02T02:27:31.479 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-02T09:38:04.464 INFO:teuthology.results:Tests finished! gathering results...
2024-02-02T09:38:04.914 INFO:scrape:Found 5 jobs
2024-02-02T09:38:04.918 INFO:scrape:Found 3 distinct failure reasons
2024-02-02T09:38:04.918 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=v16.2.7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh -k \'not test_diff_iterate\''
2024-02-02T09:38:04.918 INFO:scrape:3 jobs: ['7543176', '7543174', '7543175']
2024-02-02T09:38:04.918 INFO:scrape:suites intersection: ['1-ceph-install/pacific', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack']
2024-02-02T09:38:04.918 INFO:scrape:suites union: ['1-ceph-install/pacific', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'objectstore/bluestore-stupid', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack']
2024-02-02T09:38:04.918 INFO:scrape:
2024-02-02T09:38:04.918 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_cmpomap.sh) on smithi186 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 CLS_RBD_GTEST_FILTER=\'*:-TestClsRbd.mirror_snapshot\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cls/test_cls_cmpomap.sh'
2024-02-02T09:38:04.918 INFO:scrape:1 jobs: ['7543173']
2024-02-02T09:38:04.918 INFO:scrape:suites: ['supported-all-distro/ubuntu_latest}', 'upgrade:pacific-p2p/pacific-p2p-parallel/{point-to-point-upgrade']
2024-02-02T09:38:04.919 INFO:scrape:
2024-02-02T09:38:04.919 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json
2024-02-02T09:38:04.919 INFO:scrape:1 jobs: ['7543177']
2024-02-02T09:38:04.919 INFO:scrape:suites: ['1-ceph-install/pacific', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack']
2024-02-02T09:38:04.919 INFO:scrape:
2024-02-02T09:38:04.919 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7543173]  upgrade:pacific-p2p/pacific-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest}
-----------------------------------------------------------------
time:   00:34:48
info:   https://pulpito.ceph.com/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543173/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543173/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a92ac6ebf19e4532b9b4e0e4a845185f

    Command failed (workunit test cls/test_cls_cmpomap.sh) on smithi186 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="1" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1
    CLS_RBD_GTEST_FILTER=\'*:-TestClsRbd.mirror_snapshot\' adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/cls/test_cls_cmpomap.sh'

[7543174]  upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   07:07:23
info:   https://pulpito.ceph.com/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543174/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543174/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a732a3eefc3c40e18aed92fe6b3b45a7

    Command failed (workunit test rbd/test_librbd_python.sh) on smithi177 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=v16.2.7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh
    -k \'not test_diff_iterate\''

[7543175]  upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   05:57:25
info:   https://pulpito.ceph.com/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543175/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543175/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7239003452c94491be0a98990abe108b

    Command failed (workunit test rbd/test_librbd_python.sh) on smithi121 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=v16.2.7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh
    -k \'not test_diff_iterate\''

[7543176]  upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   06:30:23
info:   https://pulpito.ceph.com/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543176/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543176/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e7dede637f34cefb7fee095f0ed96aa

    Command failed (workunit test rbd/test_librbd_python.sh) on smithi132 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=v16.2.7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh
    -k \'not test_diff_iterate\''

[7543177]  upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   01:46:01
info:   https://pulpito.ceph.com/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543177/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-01_23:40:28-upgrade:pacific-p2p-pacific-release-distro-default-smithi/7543177/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3ff3b56c7bba4d048bff82ce0579ff7b

    Command failed on smithi073 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph osd dump --format=json'