2023-05-26T15:08:35.966 INFO:root:teuthology version: 0.0.1.dev109+ge5a3f9b
2023-05-26T15:08:35.967 INFO:teuthology.results:subset: '111/120000'
2023-05-26T15:08:35.967 INFO:teuthology.results:seed: '2582'
2023-05-26T15:39:49.700 INFO:root:teuthology version: 0.0.1.dev109+ge5a3f9b
2023-05-26T15:39:49.710 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-26T18:05:00.172 INFO:teuthology.results:Tests finished! gathering results...
2023-05-26T18:05:00.887 INFO:scrape:Found 9 jobs
2023-05-26T18:05:06.212 INFO:scrape:Found 1 distinct failure reasons
2023-05-26T18:05:06.213 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi103 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'
2023-05-26T18:05:06.213 INFO:scrape:7 jobs: ['7287163', '7287171', '7287161', '7287159', '7287165', '7287157', '7287169']
2023-05-26T18:05:06.214 INFO:scrape:suites intersection: []
2023-05-26T18:05:06.214 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.4_container_tools_rhel8}', 'upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8', 'upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04', 'workload/{ec-rados-default']
2023-05-26T18:05:06.214 INFO:scrape:
2023-05-26T18:05:06.216 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/
failed:  7
dead:    0
running: 0
waiting: 0
queued:  0
passed:  2


Fail
=================================================================
[7287157]  upgrade:pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   01:14:42
info:   http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287157/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287157/
sentry: https://sentry.ceph.com/organizations/ceph/?query=adbdc288e33c49deb014fd33bfe14110

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi012 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

[7287159]  upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:32:58
info:   http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287159/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287159/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4497540e478d41c6821a7faad86adcc5

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi084 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

[7287161]  upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:32:29
info:   http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287161/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287161/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e2a4f9047044cd68222c696a43ef3f1

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi081 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

[7287163]  upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   00:39:52
info:   http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287163/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287163/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ec59e9d732664637b180a5b4b24f610e

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi103 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

[7287165]  upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   01:13:11
info:   http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287165/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287165/
sentry: https://sentry.ceph.com/organizations/ceph/?query=28f6dd041a36429f874b4b6364703f99

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi062 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

[7287169]  upgrade:pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   01:22:41
info:   http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287169/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287169/
sentry: https://sentry.ceph.com/organizations/ceph/?query=48f7ffdcf8184897bdf7e06f5a2b6375

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi177 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

[7287171]  upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   00:43:19
info:   http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287171/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287171/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b72993cb0a8f40018c1dab5fa73041ab

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi029 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'



Pass
=================================================================
[7287167] upgrade:pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
time:   02:04:09
info:   http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287167/

[7287173] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
time:   02:23:05
info:   http://pulpito.ceph.com/teuthology-2023-05-26_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7287173/