2023-11-04T13:59:59.103 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-04T13:59:59.104 INFO:teuthology.results:subset: '111/120000' 2023-11-04T13:59:59.105 INFO:teuthology.results:seed: '1232' 2023-11-04T14:14:20.520 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-04T14:14:20.543 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-05T02:11:16.376 INFO:teuthology.results:Tests finished! gathering results... 2023-11-05T02:11:17.182 INFO:scrape:Found 9 jobs 2023-11-05T02:11:20.164 INFO:scrape:Found 2 distinct failure reasons 2023-11-05T02:11:20.165 INFO:scrape:Dead: b'2023-11-05T02:07:53.808 DEBUG:teuthology.exit:Finished running handlers' 2023-11-05T02:11:20.165 INFO:scrape:1 jobs: ['7446512'] 2023-11-05T02:11:20.166 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8'] 2023-11-05T02:11:20.166 INFO:scrape: 2023-11-05T02:11:20.167 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-11-05T02:11:20.167 INFO:scrape:2 jobs: ['7446518', '7446527'] 2023-11-05T02:11:20.168 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-11-05T02:11:20.168 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'upgrade:pacific-x/parallel/{0-random-distro$/{ubuntu_20.04}', 'workload/{ec-rados-default'] 2023-11-05T02:11:20.168 INFO:scrape: 2023-11-05T02:11:20.170 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/ failed: 2 dead: 1 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7446518] upgrade:pacific-x/parallel/{0-random-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:36:50 info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446518/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446518/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c72493c94bf4b9e85445f5725102fed Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi037 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7446527] upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:07:35 info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446527/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446527/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0eb8087f93a41c8a97b2afcc2df335e Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi050 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' Dead ================================================================= [7446512] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446512/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446512/ hit max job timeout Pass ================================================================= [7446514] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:55:34 info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446514/ [7446516] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:13:24 info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446516/ [7446520] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:02:09 info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446520/ [7446522] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:04:48 info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446522/ [7446525] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:34:29 info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446525/ [7446529] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:27:11 info: http://pulpito.ceph.com/yuriw-2023-11-04_03:35:01-upgrade:pacific-x-main-distro-default-smithi/7446529/