2023-10-07T20:39:35.154 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-10-07T20:39:35.155 INFO:teuthology.results:subset: '111/120000'
2023-10-07T20:39:35.155 INFO:teuthology.results:seed: '2505'
2023-10-07T21:04:28.100 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-10-07T21:04:28.104 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-10-08T01:21:48.087 INFO:teuthology.results:Tests finished! gathering results...
2023-10-08T01:21:49.005 INFO:scrape:Found 10 jobs
2023-10-08T01:21:52.075 WARNING:scrape:Saw end of BT but not start
2023-10-08T01:21:52.075 WARNING:scrape:Saw end of BT but not start
2023-10-08T01:21:52.076 WARNING:scrape:Saw end of BT but not start
2023-10-08T01:21:52.076 WARNING:scrape:Saw end of BT but not start
2023-10-08T01:21:52.076 WARNING:scrape:Saw end of BT but not start
2023-10-08T01:21:52.076 WARNING:scrape:Saw end of BT but not start
2023-10-08T01:21:55.451 INFO:scrape:Found 3 distinct failure reasons
2023-10-08T01:21:55.452 INFO:scrape:Failure: 
2023-10-08T01:21:55.452 INFO:scrape:7 jobs: ['7416914', '7416922', '7416906', '7416908', '7416912', '7416916', '7416904']
2023-10-08T01:21:55.452 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}']
2023-10-08T01:21:55.452 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0', 'upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04']
2023-10-08T01:21:55.452 INFO:scrape:
2023-10-08T01:21:55.453 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh
2023-10-08T01:21:55.453 INFO:scrape:2 jobs: ['7416920', '7416910']
2023-10-08T01:21:55.453 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']
2023-10-08T01:21:55.453 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'workload/{ec-rados-default']
2023-10-08T01:21:55.453 INFO:scrape:
2023-10-08T01:21:55.453 INFO:scrape:Failure: "1696713365.1348658 mon.a (mon.0) 69 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
2023-10-08T01:21:55.453 INFO:scrape:1 jobs: ['7416918']
2023-10-08T01:21:55.453 INFO:scrape:suites: ['-', '1-ceph-install/quincy', '2', 'objectstore/filestore-xfs', 'start}', 'ubuntu_20.04}', 'upgrade', 'upgrade:quincy-x/filestore-remove-check/{0-cluster/{openstack']
2023-10-08T01:21:55.454 INFO:scrape:
2023-10-08T01:21:55.455 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/
failed:  10
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7416904]  upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   01:16:17
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416904/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416904/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7ca9d331304d4f3abdc4cfadaae7c33d

[7416906]  upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:58:34
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416906/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416906/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3e9db5a58adc4252a962ac00d60f6a8a

[7416908]  upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:18:51
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416908/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416908/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bac31bf729e5464cad467977ec292f52

[7416910]  upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   04:14:59
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416910/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416910/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6515080f6e8b4dfc9ad30227aeceeced

    Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi003 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh'

[7416912]  upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   01:12:17
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416912/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416912/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e04e345bf8e249efa4513cf904c0936c

[7416914]  upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:08:30
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416914/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416914/
sentry: https://sentry.ceph.com/organizations/ceph/?query=870711075bdd42c6a107c55b6bb493e8

[7416916]  upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   00:22:41
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416916/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416916/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2f59726216c34e33ba8d8d4829dfc877

[7416918]  upgrade:quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_20.04}
-----------------------------------------------------------------
time:   00:09:07
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416918/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416918/

    "1696713365.1348658 mon.a (mon.0) 69 : cluster [WRN] Health check failed: 1
    pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in
    cluster log

[7416920]  upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   04:12:36
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416920/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416920/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5302ba019b6a47459b098bf7c9971cdf

    Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi006 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh'

[7416922]  upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   01:49:21
info:   http://pulpito.ceph.com/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416922/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-07_14:03:29-upgrade:quincy-x-main-distro-default-smithi/7416922/
sentry: https://sentry.ceph.com/organizations/ceph/?query=78f9335822c9445fbd49090cd86f4c17