2023-01-16T17:24:02.688 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-16T17:24:02.689 INFO:teuthology.results:subset: '111/120000' 2023-01-16T17:24:02.689 INFO:teuthology.results:seed: '4896' 2023-01-16T18:39:23.314 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-16T18:39:23.318 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-01-17T06:39:24.494 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-01-17T06:39:27.886 INFO:scrape:Found 29 jobs 2023-01-17T06:39:28.846 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143278/teuthology.log 2023-01-17T06:39:30.406 INFO:scrape:Found 6 distinct failure reasons 2023-01-17T06:39:30.406 INFO:scrape:Failure: Command failed on smithi073 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.17-1bionic ceph-mds=15.2.17-1bionic ceph-mgr=15.2.17-1bionic ceph-common=15.2.17-1bionic ceph-fuse=15.2.17-1bionic ceph-test=15.2.17-1bionic radosgw=15.2.17-1bionic python3-rados=15.2.17-1bionic python3-rgw=15.2.17-1bionic python3-cephfs=15.2.17-1bionic python3-rbd=15.2.17-1bionic libcephfs2=15.2.17-1bionic librados2=15.2.17-1bionic librbd1=15.2.17-1bionic rbd-fuse=15.2.17-1bionic' 2023-01-17T06:39:30.407 INFO:scrape:5 jobs: ['7143279', '7143290', '7143288', '7143280', '7143287'] 2023-01-17T06:39:30.407 INFO:scrape:suites intersection: [] 2023-01-17T06:39:30.407 INFO:scrape:suites union: ['0-roles', '1-ceph-install/octopus', '1-octopus-install/octopus', '1-start', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-first-half-tasks/rbd-import-export', '2-partial-upgrade/firsthalf', '3-stress-tasks/{radosbench', '3-thrash/default', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-second-half-tasks/rbd-import-export', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-final-workload', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04'] 2023-01-17T06:39:30.407 INFO:scrape: 2023-01-17T06:39:30.407 INFO:scrape:Failure: HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubuntu/bionic/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) 2023-01-17T06:39:30.407 INFO:scrape:12 jobs: ['7143267', '7143266', '7143265', '7143274', '7143272', '7143270', '7143268', '7143264', '7143269', '7143273', '7143271', '7143263'] 2023-01-17T06:39:30.408 INFO:scrape:suites intersection: [] 2023-01-17T06:39:30.408 INFO:scrape:suites union: ['0-roles', '0-start', '1-ceph-install/octopus', '1-octopus-install/octopus', '1-start', '1-tasks', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-partial-upgrade/firsthalf', '2-workload/{rgw_ragweed_prepare}', '3-stress-tasks/{radosbench', '3-thrash/default', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '3.1-objectstore/bluestore-hybrid', '4-ec-workload/{rados-ec-workload', '4-pacific', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', '4-workload/{radosbench', '5-final-workload/{rgw', '5-finish-upgrade', '6-pacific', '7-final-workload', '8-final-workload/{rbd-python', 'centos_latest}', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api}}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade-sequence', 'upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04', 'upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04', 'upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8', 'upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04', 'upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04', 'workload/{ec-rados-default'] 2023-01-17T06:39:30.408 INFO:scrape: 2023-01-17T06:39:30.408 INFO:scrape:Failure: Command failed on smithi145 with status 5: 'sudo systemctl stop ceph-75ab37b0-95cc-11ed-a3e6-001a4aab830c@mon.a' 2023-01-17T06:39:30.408 INFO:scrape:8 jobs: ['7143284', '7143291', '7143277', '7143276', '7143285', '7143283', '7143282', '7143289'] 2023-01-17T06:39:30.408 INFO:scrape:suites intersection: [] 2023-01-17T06:39:30.409 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8', 'upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8', 'workload/{ec-rados-default'] 2023-01-17T06:39:30.409 INFO:scrape: 2023-01-17T06:39:30.409 INFO:scrape:Failure: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) 2023-01-17T06:39:30.409 INFO:scrape:1 jobs: ['7143281'] 2023-01-17T06:39:30.409 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04', 'workload/{ec-rados-default'] 2023-01-17T06:39:30.409 INFO:scrape: 2023-01-17T06:39:30.409 INFO:scrape:Dead: None 2023-01-17T06:39:30.409 INFO:scrape:1 jobs: ['7143278'] 2023-01-17T06:39:30.410 INFO:scrape: 2023-01-17T06:39:30.410 INFO:scrape:Failure: 502 Server Error: Bad Gateway for url: https://chacra.ceph.com/repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/repo 2023-01-17T06:39:30.410 INFO:scrape:1 jobs: ['7143275'] 2023-01-17T06:39:30.410 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0', 'workload/{ec-rados-default'] 2023-01-17T06:39:30.410 INFO:scrape: 2023-01-17T06:39:30.412 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/ failed: 27 dead: 0 running: 0 waiting: 1 queued: 0 passed: 1 Fail ================================================================= [7143263] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} ----------------------------------------------------------------- time: 00:08:41 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143263/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143263/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a37aa2055c240218665cfb20c4a331a HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/cent os/8/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143264] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:06:59 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143264/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143264/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bdd385f202fc43a9980d4e1f51c6160c HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubun tu/bionic/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143265] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:07:03 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143265/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143265/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5707f3f8306241b49be15038bb95284f HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubun tu/bionic/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143266] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:09:03 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143266/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143266/ sentry: https://sentry.ceph.com/organizations/ceph/?query=871b205862eb47fcbaceb127e29d1366 HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/cent os/8/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143267] upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:06:54 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143267/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143267/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad5d7bd202aa463ba316e1390c7d3e60 HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubun tu/bionic/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143268] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:06:58 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143268/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143268/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd1d1765d2bc495498f2a8db76ea85f1 HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubun tu/focal/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143269] upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:06:43 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143269/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143269/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9711cfcca7944af49c37a9a35fdaf975 HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubun tu/bionic/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143270] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:09:45 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143270/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143270/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b363960884e4857844856cef489c055 HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/cent os/8/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143271] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} ----------------------------------------------------------------- time: 00:08:44 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143271/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143271/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0177ba78a3a2493fb426798fbe264e5a HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/cent os/8/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143272] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:06:55 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143272/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143272/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3716d4f3a0a84089b1db709e4c30d799 HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubun tu/bionic/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143273] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:06:57 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143273/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143273/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8321926ffd840f1b353730dcb2cac61 HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubun tu/bionic/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143274] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:07:02 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143274/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143274/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85af61f4e73b4118b1b15a1543d88575 HTTPSConnectionPool(host='chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubun tu/focal/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7143275] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:07:20 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143275/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143275/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9a6bff3857b4227941d72273af4eb8b 502 Server Error: Bad Gateway for url: https://chacra.ceph.com/repos/ceph/o ctopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/re po [7143276] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:11:30 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143276/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143276/ sentry: https://sentry.ceph.com/organizations/ceph/?query=918050a35f8b4cfc92723ea8d878c57e Command failed on smithi082 with status 5: 'sudo systemctl stop ceph-7767afaa-95c9-11ed-a3e6-001a4aab830c@mon.a' [7143277] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:11:22 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143277/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143277/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04da53c809e8477dbf78353e5caee947 Command failed on smithi149 with status 5: 'sudo systemctl stop ceph-9c5b2b98-95c9-11ed-a3e6-001a4aab830c@mon.a' [7143279] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:07:20 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143279/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143279/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4dcf316265cd413d9744e36a89355c17 Command failed on smithi073 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.17-1bionic ceph-mds=15.2.17-1bionic ceph-mgr=15.2.17-1bionic ceph-common=15.2.17-1bionic ceph-fuse=15.2.17-1bionic ceph- test=15.2.17-1bionic radosgw=15.2.17-1bionic python3-rados=15.2.17-1bionic python3-rgw=15.2.17-1bionic python3-cephfs=15.2.17-1bionic python3-rbd=15.2.17-1bionic libcephfs2=15.2.17-1bionic librados2=15.2.17-1bionic librbd1=15.2.17-1bionic rbd-fuse=15.2.17-1bionic' [7143280] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:07:49 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143280/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143280/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89377c53cc4d457c85fb04eafcfb7693 Command failed on smithi078 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.17-1bionic ceph-mds=15.2.17-1bionic ceph-mgr=15.2.17-1bionic ceph-common=15.2.17-1bionic ceph-fuse=15.2.17-1bionic ceph- test=15.2.17-1bionic radosgw=15.2.17-1bionic python3-rados=15.2.17-1bionic python3-rgw=15.2.17-1bionic python3-cephfs=15.2.17-1bionic python3-rbd=15.2.17-1bionic libcephfs2=15.2.17-1bionic librados2=15.2.17-1bionic librbd1=15.2.17-1bionic rbd-fuse=15.2.17-1bionic' [7143281] upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:04:52 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143281/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143281/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fdc88515fef9428b84f2b4719e7f324a ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) [7143282] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:11:55 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143282/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143282/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68f3cf7a4ed04ccdb7f569e79c1811e8 Command failed on smithi082 with status 5: 'sudo systemctl stop ceph-e98b450e-95cb-11ed-a3e6-001a4aab830c@mon.a' [7143283] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:07:28 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143283/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143283/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9e059cbdf164196b8194c28ecc63ca7 Command failed on smithi149 with status 5: 'sudo systemctl stop ceph-40c44c44-95cc-11ed-a3e6-001a4aab830c@mon.a' [7143284] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:11:29 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143284/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143284/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83ecf272985b4f299aa3fc2cd0635ce8 Command failed on smithi145 with status 5: 'sudo systemctl stop ceph-75ab37b0-95cc-11ed-a3e6-001a4aab830c@mon.a' [7143285] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:11:15 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143285/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143285/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d2db091ea5c4d9793439e0b082b2785 Command failed on smithi073 with status 5: 'sudo systemctl stop ceph-76aae200-95cc-11ed-a3e6-001a4aab830c@mon.a' [7143287] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:06:51 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143287/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143287/ sentry: https://sentry.ceph.com/organizations/ceph/?query=770c1fa379bc4f4fa78038c161c6f49e Command failed on smithi110 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.17-1bionic ceph-mds=15.2.17-1bionic ceph-mgr=15.2.17-1bionic ceph-common=15.2.17-1bionic ceph-fuse=15.2.17-1bionic ceph- test=15.2.17-1bionic radosgw=15.2.17-1bionic python3-rados=15.2.17-1bionic python3-rgw=15.2.17-1bionic python3-cephfs=15.2.17-1bionic python3-rbd=15.2.17-1bionic libcephfs2=15.2.17-1bionic librados2=15.2.17-1bionic librbd1=15.2.17-1bionic rbd-fuse=15.2.17-1bionic' [7143288] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:07:47 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143288/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143288/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6bbff69fc3a3461fa4318195341ec27c Command failed on smithi151 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.17-1bionic ceph-mds=15.2.17-1bionic ceph-mgr=15.2.17-1bionic ceph-common=15.2.17-1bionic ceph-fuse=15.2.17-1bionic ceph- test=15.2.17-1bionic radosgw=15.2.17-1bionic python3-rados=15.2.17-1bionic python3-rgw=15.2.17-1bionic python3-cephfs=15.2.17-1bionic python3-rbd=15.2.17-1bionic libcephfs2=15.2.17-1bionic librados2=15.2.17-1bionic librbd1=15.2.17-1bionic rbd-fuse=15.2.17-1bionic' [7143289] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:25:57 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143289/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143289/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79c317f62316498cb9e4c29f41b31de5 Command failed on smithi073 with status 1: 'sudo yum -y install cephadm' [7143290] upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:16:37 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143290/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143290/ sentry: https://sentry.ceph.com/organizations/ceph/?query=82953df967ed46aa9fab1899ed3f231f Command failed on smithi132 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.17-1bionic cephadm=15.2.17-1bionic ceph-mds=15.2.17-1bionic ceph- mgr=15.2.17-1bionic ceph-common=15.2.17-1bionic ceph-fuse=15.2.17-1bionic ceph-test=15.2.17-1bionic radosgw=15.2.17-1bionic python3-rados=15.2.17-1bionic python3-rgw=15.2.17-1bionic python3-cephfs=15.2.17-1bionic python3-rbd=15.2.17-1bionic libcephfs2=15.2.17-1bionic libcephfs-dev=15.2.17-1bionic librados2=15.2.17-1bionic librbd1=15.2.17-1bionic rbd-fuse=15.2.17-1bionic' [7143291] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:25:10 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143291/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143291/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6650b834b640480a88e8f7e02430890a Command failed on smithi110 with status 1: 'sudo yum -y install ceph-test' Waiting ================================================================= [7143278] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143278/ Pass ================================================================= [7143286] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:22:58 info: http://pulpito.ceph.com/yuriw-2023-01-16_15:50:36-upgrade:octopus-x-pacific_16.2.11_RC6.6-distro-default-smithi/7143286/