2023-10-15T21:16:13.399 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-15T21:16:13.400 INFO:teuthology.results:subset: '111/120000' 2023-10-15T21:16:13.401 INFO:teuthology.results:seed: '5655' 2023-10-15T21:26:09.332 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-15T21:26:09.337 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-16T02:24:32.223 INFO:teuthology.results:Tests finished! gathering results... 2023-10-16T02:24:32.929 INFO:scrape:Found 10 jobs 2023-10-16T02:24:33.606 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427089/teuthology.log 2023-10-16T02:24:34.182 INFO:scrape:Found 4 distinct failure reasons 2023-10-16T02:24:34.183 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-10-16T02:24:34.183 INFO:scrape:2 jobs: ['7427087', '7427092'] 2023-10-16T02:24:34.183 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-10-16T02:24:34.184 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'upgrade:quincy-x/parallel/{0-random-distro$/{ubuntu_20.04}', 'workload/{ec-rados-default'] 2023-10-16T02:24:34.184 INFO:scrape: 2023-10-16T02:24:34.185 INFO:scrape:Failure: "1697406170.1497488 mon.a (mon.0) 63 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-10-16T02:24:34.185 INFO:scrape:1 jobs: ['7427091'] 2023-10-16T02:24:34.185 INFO:scrape:suites: ['-', '1-ceph-install/quincy', '2', 'objectstore/filestore-xfs', 'start}', 'ubuntu_20.04}', 'upgrade', 'upgrade:quincy-x/filestore-remove-check/{0-cluster/{openstack'] 2023-10-16T02:24:34.186 INFO:scrape: 2023-10-16T02:24:34.186 INFO:scrape:Dead: None 2023-10-16T02:24:34.186 INFO:scrape:1 jobs: ['7427089'] 2023-10-16T02:24:34.187 INFO:scrape: 2023-10-16T02:24:34.187 INFO:scrape:Failure: timeout expired in wait_until_healthy 2023-10-16T02:24:34.188 INFO:scrape:1 jobs: ['7427093'] 2023-10-16T02:24:34.188 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04'] 2023-10-16T02:24:34.188 INFO:scrape: 2023-10-16T02:24:34.189 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/ failed: 4 dead: 1 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7427087] upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:11:58 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427087/ log: http://qa-proxy.ceph.com/teuthology/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427087/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e2aae34342a4fec9ef74b74c0da5085 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi078 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7427091] upgrade:quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_20.04} ----------------------------------------------------------------- time: 00:08:58 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427091/ log: http://qa-proxy.ceph.com/teuthology/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427091/ "1697406170.1497488 mon.a (mon.0) 63 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7427092] upgrade:quincy-x/parallel/{0-random-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:48:43 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427092/ log: http://qa-proxy.ceph.com/teuthology/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427092/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a25f055b99a84aa79191f233f68d521d Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi110 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7427093] upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:43:58 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427093/ log: http://qa-proxy.ceph.com/teuthology/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427093/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2bee698909a41bebc016c1c2bca645f timeout expired in wait_until_healthy Dead ================================================================= [7427089] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427089/ log: http://qa-proxy.ceph.com/teuthology/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427089/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7427084] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:21:20 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427084/ [7427085] upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:55:14 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427085/ [7427086] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:27:05 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427086/ [7427088] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:08:43 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427088/ [7427090] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:13:45 info: http://pulpito.ceph.com/lflores-2023-10-13_22:46:36-upgrade:quincy-x-main-distro-default-smithi/7427090/