2023-10-09T21:33:28.364 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-09T21:33:28.365 INFO:teuthology.results:subset: '111/120000' 2023-10-09T21:33:28.365 INFO:teuthology.results:seed: '6472' 2023-10-09T21:43:48.425 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-09T21:43:48.428 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-10T02:08:08.126 INFO:teuthology.results:Tests finished! gathering results... 2023-10-10T02:08:08.740 INFO:scrape:Found 7 jobs 2023-10-10T02:08:09.735 INFO:scrape:Found 2 distinct failure reasons 2023-10-10T02:08:09.735 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-10-10T02:08:09.735 INFO:scrape:1 jobs: ['7419090'] 2023-10-10T02:08:09.736 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'workload/{ec-rados-default'] 2023-10-10T02:08:09.736 INFO:scrape: 2023-10-10T02:08:09.736 INFO:scrape:Failure: ceph version 17.2.6-1340.gb00ee4ed was not installed, found 17.2.6-1303.gb2903cb0.el8. 2023-10-10T02:08:09.736 INFO:scrape:2 jobs: ['7419091', '7419089'] 2023-10-10T02:08:09.736 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2023-10-10T02:08:09.736 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun'] 2023-10-10T02:08:09.736 INFO:scrape: 2023-10-10T02:08:09.737 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [7419089] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:54:45 info: http://pulpito.ceph.com/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419089/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419089/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89644c734363464fa440bc269ac641a6 ceph version 17.2.6-1340.gb00ee4ed was not installed, found 17.2.6-1303.gb2903cb0.el8. [7419090] upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:19:28 info: http://pulpito.ceph.com/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419090/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419090/ sentry: https://sentry.ceph.com/organizations/ceph/?query=526bd11338ef4dc385818b9bc46e204c Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi115 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7419091] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:59:00 info: http://pulpito.ceph.com/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419091/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419091/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4014f95d52ba4ad387b36c7479cbb1d2 ceph version 17.2.6-1340.gb00ee4ed was not installed, found 17.2.6-1303.gb2903cb0.el8. Pass ================================================================= [7419087] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:08:10 info: http://pulpito.ceph.com/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419087/ [7419088] upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:51:44 info: http://pulpito.ceph.com/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419088/ [7419092] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:02:15 info: http://pulpito.ceph.com/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419092/ [7419093] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:15:53 info: http://pulpito.ceph.com/yuriw-2023-10-09_20:41:59-upgrade:quincy-x-main-distro-default-smithi/7419093/