2023-09-30T08:20:34.007 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-30T08:20:34.009 INFO:teuthology.results:subset: '111/120000'
2023-09-30T08:20:34.009 INFO:teuthology.results:seed: '9689'
2023-09-30T08:30:04.470 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-30T08:30:04.475 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-30T12:47:24.386 INFO:teuthology.results:Tests finished! gathering results...
2023-09-30T12:47:25.176 INFO:scrape:Found 9 jobs
2023-09-30T12:47:30.236 INFO:scrape:Found 2 distinct failure reasons
2023-09-30T12:47:30.237 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh
2023-09-30T12:47:30.237 INFO:scrape:2 jobs: ['7407166', '7407178']
2023-09-30T12:47:30.237 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']
2023-09-30T12:47:30.237 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools}', 'workload/{ec-rados-default']
2023-09-30T12:47:30.238 INFO:scrape:
2023-09-30T12:47:30.238 INFO:scrape:Failure: 
2023-09-30T12:47:30.238 INFO:scrape:7 jobs: ['7407157', '7407160', '7407163', '7407169', '7407172', '7407175', '7407181']
2023-09-30T12:47:30.238 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}']
2023-09-30T12:47:30.238 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04']
2023-09-30T12:47:30.238 INFO:scrape:
2023-09-30T12:47:30.239 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/
failed:  9
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7407157]  upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   01:11:47
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407157/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407157/
sentry: https://sentry.ceph.com/organizations/ceph/?query=524674725de94e97aa6afde0da77d415

[7407160]  upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:41:56
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407160/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407160/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6e4bd9651ca247978780527e79338812

[7407163]  upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:33:36
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407163/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407163/
sentry: https://sentry.ceph.com/organizations/ceph/?query=79527e12e86b4d3cadc9e539096181b8

[7407166]  upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   04:09:31
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407166/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407166/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0f0e6f50635c415b8fa185d3f0253e5c

    Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi067 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh'

[7407169]  upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   01:13:08
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407169/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407169/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5acfb31cfb5f4349a43d04f2c7f7622e

[7407172]  upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:17:21
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407172/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407172/
sentry: https://sentry.ceph.com/organizations/ceph/?query=444c96fe6ac044c7b59e8e9049acca7b

[7407175]  upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   00:24:39
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407175/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407175/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c3419f20899d458d959883f9b0f731bf

[7407178]  upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   04:08:31
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407178/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407178/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e761c58c2b5a475e88a912c6c72c1940

    Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi093 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh'

[7407181]  upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:28:19
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407181/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:44:04-upgrade:pacific-x-main-distro-default-smithi/7407181/
sentry: https://sentry.ceph.com/organizations/ceph/?query=87a12a0915ad4b7e97bd07bdfb75d03a