2023-03-03T18:48:00.987 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-03T18:48:00.988 INFO:teuthology.results:subset: '111/120000' 2023-03-03T18:48:00.988 INFO:teuthology.results:seed: '6564' 2023-03-03T19:10:57.322 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-03T19:10:57.325 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-03T21:25:08.238 INFO:teuthology.results:Tests finished! gathering results... 2023-03-03T21:25:09.037 INFO:scrape:Found 10 jobs 2023-03-03T21:25:11.269 INFO:scrape:Found 1 distinct failure reasons 2023-03-03T21:25:11.271 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-03-03T21:25:11.271 INFO:scrape:4 jobs: ['7192577', '7192593', '7192571', '7192565'] 2023-03-03T21:25:11.275 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2023-03-03T21:25:11.276 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04'] 2023-03-03T21:25:11.276 INFO:scrape: 2023-03-03T21:25:11.278 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7192565] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:19:50 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192565/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192565/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3af9dae78e3149db9068d0ca11234ff5 Command failed (workunit test cls/test_cls_rbd.sh) on smithi171 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7192571] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:21:50 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192571/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192571/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cadf0091bd81426085c8d01ca0957e65 Command failed (workunit test cls/test_cls_rbd.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7192577] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:42 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192577/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192577/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d171e44c80ef430281f5252a17fb0c5e Command failed (workunit test cls/test_cls_rbd.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7192593] upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:27:35 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192593/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192593/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a9fff5e65f444d98b1bac5552f1fbea8 Command failed (workunit test cls/test_cls_rbd.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' Pass ================================================================= [7192568] upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:25:08 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192568/ [7192574] upgrade:quincy-x/parallel/{0-random-distro$/{centos_8.stream_container_tools} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:25:41 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192574/ [7192580] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:55:56 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192580/ [7192583] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:03:05 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192583/ [7192586] upgrade:quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_20.04} time: 00:08:38 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192586/ [7192590] upgrade:quincy-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:25:18 info: http://pulpito.ceph.com/yuriw-2023-03-03_17:40:19-upgrade:quincy-x-reef-distro-default-smithi/7192590/