2023-10-21T10:20:06.753 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-21T10:20:06.754 INFO:teuthology.results:subset: '111/120000' 2023-10-21T10:20:06.755 INFO:teuthology.results:seed: '7983' 2023-10-21T10:28:56.221 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-21T10:28:56.225 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-21T14:50:16.681 INFO:teuthology.results:Tests finished! gathering results... 2023-10-21T14:50:17.555 INFO:scrape:Found 9 jobs 2023-10-21T14:50:18.868 INFO:scrape:Found 4 distinct failure reasons 2023-10-21T14:50:18.869 INFO:scrape:Failure: Command failed on smithi040 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.14-92-ge99bbbdf-1focal cephadm=16.2.14-92-ge99bbbdf-1focal ceph-mds=16.2.14-92-ge99bbbdf-1focal ceph-mgr=16.2.14-92-ge99bbbdf-1focal ceph-common=16.2.14-92-ge99bbbdf-1focal ceph-fuse=16.2.14-92-ge99bbbdf-1focal ceph-test=16.2.14-92-ge99bbbdf-1focal radosgw=16.2.14-92-ge99bbbdf-1focal python3-rados=16.2.14-92-ge99bbbdf-1focal python3-rgw=16.2.14-92-ge99bbbdf-1focal python3-cephfs=16.2.14-92-ge99bbbdf-1focal python3-rbd=16.2.14-92-ge99bbbdf-1focal libcephfs2=16.2.14-92-ge99bbbdf-1focal libcephfs-dev=16.2.14-92-ge99bbbdf-1focal librados2=16.2.14-92-ge99bbbdf-1focal librbd1=16.2.14-92-ge99bbbdf-1focal rbd-fuse=16.2.14-92-ge99bbbdf-1focal' 2023-10-21T14:50:18.869 INFO:scrape:1 jobs: ['7433928'] 2023-10-21T14:50:18.870 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04'] 2023-10-21T14:50:18.870 INFO:scrape: 2023-10-21T14:50:18.870 INFO:scrape:Failure: HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) 2023-10-21T14:50:18.871 INFO:scrape:5 jobs: ['7433911', '7433921', '7433924', '7433917', '7433920'] 2023-10-21T14:50:18.871 INFO:scrape:suites intersection: [] 2023-10-21T14:50:18.872 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'workload/{ec-rados-default'] 2023-10-21T14:50:18.872 INFO:scrape: 2023-10-21T14:50:18.873 INFO:scrape:Failure: ['Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: '] 2023-10-21T14:50:18.873 INFO:scrape:1 jobs: ['7433913'] 2023-10-21T14:50:18.873 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04'] 2023-10-21T14:50:18.874 INFO:scrape: 2023-10-21T14:50:18.874 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-10-21T14:50:18.875 INFO:scrape:1 jobs: ['7433926'] 2023-10-21T14:50:18.875 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'workload/{ec-rados-default'] 2023-10-21T14:50:18.875 INFO:scrape: 2023-10-21T14:50:18.877 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/ failed: 7 dead: 1 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7433911] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:33:23 info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433911/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433911/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50a118bb7d7f4513b0e34184a7be211d HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F8%2Fx86_64&ref=pacific (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7433917] upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:12:06 info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433917/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433917/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0b2dc40d9234f698dff66f89fc9a7ca HTTPSConnectionPool(host='2.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/pacific/e99bbbdfb436c14c315eeca61e4f13b6c491 e5fb/centos/8/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7433920] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:10:04 info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433920/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433920/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb76064ea7ab4719bf997e1958bc5257 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F8%2Fx86_64&ref=pacific (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7433921] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:12:39 info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433921/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433921/ sentry: https://sentry.ceph.com/organizations/ceph/?query=371472ec3ce143a3a5f9eca40f44d515 HTTPSConnectionPool(host='2.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/pacific/e99bbbdfb436c14c315eeca61e4f13b6c491 e5fb/centos/8/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7433924] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:11:54 info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433924/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca30b29c9b1e4ebda715ed3924223733 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F8%2Fx86_64&ref=pacific (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7433926] upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:16:13 info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433926/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af539e496e8242d29138421b60966f33 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi156 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7433928] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:06:55 info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433928/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433928/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ed42e802f1b4785b79472b25e565350 Command failed on smithi040 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.14-92-ge99bbbdf-1focal cephadm=16.2.14-92-ge99bbbdf-1focal ceph- mds=16.2.14-92-ge99bbbdf-1focal ceph-mgr=16.2.14-92-ge99bbbdf-1focal ceph- common=16.2.14-92-ge99bbbdf-1focal ceph-fuse=16.2.14-92-ge99bbbdf-1focal ceph-test=16.2.14-92-ge99bbbdf-1focal radosgw=16.2.14-92-ge99bbbdf-1focal python3-rados=16.2.14-92-ge99bbbdf-1focal python3-rgw=16.2.14-92-ge99bbbdf-1focal python3-cephfs=16.2.14-92-ge99bbbdf-1focal python3-rbd=16.2.14-92-ge99bbbdf-1focal libcephfs2=16.2.14-92-ge99bbbdf-1focal libcephfs- dev=16.2.14-92-ge99bbbdf-1focal librados2=16.2.14-92-ge99bbbdf-1focal librbd1=16.2.14-92-ge99bbbdf-1focal rbd-fuse=16.2.14-92-ge99bbbdf-1focal' Dead ================================================================= [7433913] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:03:23 info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433913/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433913/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4900cfd88014c2a87c317dd74d3cb13 ['Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: '] Pass ================================================================= [7433915] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:25:39 info: http://pulpito.ceph.com/yuriw-2023-10-20_18:44:32-upgrade:pacific-x-main-distro-default-smithi/7433915/