2023-09-05T15:04:50.069 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-09-05T15:04:50.071 INFO:teuthology.results:subset: '100/101' 2023-09-05T15:04:50.071 INFO:teuthology.results:seed: '4659' 2023-09-05T15:09:54.397 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-09-05T15:09:54.402 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-05T19:57:14.242 INFO:teuthology.results:Tests finished! gathering results... 2023-09-05T19:57:15.042 INFO:scrape:Found 10 jobs 2023-09-05T19:57:15.691 INFO:scrape:Found 1 distinct failure reasons 2023-09-05T19:57:15.692 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-09-05T19:57:15.693 INFO:scrape:2 jobs: ['7388668', '7388673'] 2023-09-05T19:57:15.693 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-09-05T19:57:15.693 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:quincy-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'workload/{ec-rados-default'] 2023-09-05T19:57:15.694 INFO:scrape: 2023-09-05T19:57:15.695 INFO:teuthology.results:Sending results to akupczyk@redhat.com: Test Run: akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7388668] upgrade:quincy-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:42:34 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388668/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388668/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb7aefbe24a04f0c88495fa8546b01c2 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi029 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7388673] upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:15:02 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388673/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388673/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d1a859bb0e9420aafc6593b1c6d8775 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi005 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' Pass ================================================================= [7388664] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:07:26 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388664/ [7388665] upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:33:39 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388665/ [7388666] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:25:07 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388666/ [7388667] upgrade:quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_20.04} time: 00:09:27 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388667/ [7388669] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:57:54 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388669/ [7388670] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:16:17 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388670/ [7388671] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:09:14 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388671/ [7388672] upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:43:46 info: http://pulpito.ceph.com/akupczyk-2023-09-05_14:00:25-upgrade:quincy-x-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7388672/