2024-04-10T14:27:28.043 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-10T14:27:28.043 INFO:teuthology.results:subset: '111/120000' 2024-04-10T14:27:28.043 INFO:teuthology.results:seed: '4372' 2024-04-10T19:39:35.391 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-10T19:39:35.411 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-11T03:22:18.414 INFO:teuthology.results:Tests finished! gathering results... 2024-04-11T03:22:31.163 INFO:scrape:Found 293 jobs 2024-04-11T03:22:34.872 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651065/teuthology.log 2024-04-11T03:22:45.054 INFO:scrape:Found 9 distinct failure reasons 2024-04-11T03:22:45.054 INFO:scrape:Failure: HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) 2024-04-11T03:22:45.054 INFO:scrape:235 jobs 2024-04-11T03:22:45.054 INFO:scrape: 2024-04-11T03:22:45.055 INFO:scrape:Failure: ['Failed to download key at http://download.ceph.com/keys/autobuild.asc: Request failed: ', 'Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: '] 2024-04-11T03:22:45.055 INFO:scrape:2 jobs: ['7650717', '7650715'] 2024-04-11T03:22:45.055 INFO:scrape:suites intersection: ['rados'] 2024-04-11T03:22:45.055 INFO:scrape:suites union: ['cluster/{12-osds', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/cache-fs-trunc', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}'] 2024-04-11T03:22:45.055 INFO:scrape: 2024-04-11T03:22:45.055 INFO:scrape:Failure: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-11T03:22:45.055 INFO:scrape:7 jobs: ['7650781', '7650992', '7650856', '7650922', '7650818', '7650745', '7651055'] 2024-04-11T03:22:45.055 INFO:scrape:suites intersection: ['ceph', 'clusters/{openstack', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashosds-health', 'three-plus-one}'] 2024-04-11T03:22:45.056 INFO:scrape:suites union: ['0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-install/octopus', '1-install/pacific', '1-install/quincy', '1-install/reef', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] 2024-04-11T03:22:45.056 INFO:scrape: 2024-04-11T03:22:45.056 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5074d4516d566e9d8b6aec912f26afd099de101 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2024-04-11T03:22:45.056 INFO:scrape:1 jobs: ['7650664'] 2024-04-11T03:22:45.056 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] 2024-04-11T03:22:45.056 INFO:scrape: 2024-04-11T03:22:45.056 INFO:scrape:Failure: "2024-04-10T15:47:39.072303+0000 mon.a (mon.0) 536 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2024-04-11T03:22:45.056 INFO:scrape:2 jobs: ['7650700', '7650687'] 2024-04-11T03:22:45.056 INFO:scrape:suites intersection: ['mon_election/classic'] 2024-04-11T03:22:45.056 INFO:scrape:suites union: ['0-nvme-loop', '0-start', '1-tasks', 'agent/off', 'fixed-2', 'mon_election/classic', 'rados/cephadm/smoke/{0-distro/centos_9.stream', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'start}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-04-11T03:22:45.057 INFO:scrape: 2024-04-11T03:22:45.057 INFO:scrape:Failure: Command failed on smithi087 with status 1: 'sudo yum -y install ceph-radosgw' 2024-04-11T03:22:45.057 INFO:scrape:3 jobs: ['7650669', '7650712', '7650713'] 2024-04-11T03:22:45.057 INFO:scrape:suites intersection: ['mon_election/connectivity', 'rados'] 2024-04-11T03:22:45.057 INFO:scrape:suites union: ['0-size-min-size-overrides/3-size-2-min-size', '1-install/reef', 'backoff/normal', 'ceph', 'clusters/{fixed-4', 'clusters/{openstack', 'd-balancer/on', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/many', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/multimon/{clusters/9', 'rados/thrash-erasure-code/{ceph', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'tasks/mon_clock_with_skews}', 'thrashers/careful', 'thrashers/default', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/ec-small-objects-fast-read}'] 2024-04-11T03:22:45.057 INFO:scrape: 2024-04-11T03:22:45.057 INFO:scrape:Dead: b'2024-04-11T03:17:43.046 DEBUG:teuthology.exit:Finished running handlers' 2024-04-11T03:22:45.057 INFO:scrape:2 jobs: ['7650684', '7650670'] 2024-04-11T03:22:45.057 INFO:scrape:suites intersection: [] 2024-04-11T03:22:45.057 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-flag}', 'mon_election/classic', 'rados', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-11T03:22:45.058 INFO:scrape: 2024-04-11T03:22:45.058 INFO:scrape:Dead: None 2024-04-11T03:22:45.058 INFO:scrape:11 jobs: ['7651065', '7650831', '7651023', '7651011', '7650725', '7651070', '7650755', '7650756', '7650771', '7651063', '7651078'] 2024-04-11T03:22:45.058 INFO:scrape: 2024-04-11T03:22:45.058 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5074d4516d566e9d8b6aec912f26afd099de101 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-04-11T03:22:45.058 INFO:scrape:1 jobs: ['7650646'] 2024-04-11T03:22:45.058 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_cephadm}'] 2024-04-11T03:22:45.058 INFO:scrape: 2024-04-11T03:22:45.061 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/ failed: 242 dead: 22 running: 0 waiting: 0 queued: 0 passed: 29 Fail ================================================================= [7650646] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:13:49 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650646/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650646/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dbe7353c1f374915a3bc26087ce4aa55 Command failed (workunit test cephadm/test_cephadm.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5074d4516d566e9d8b6aec912f26afd099de101 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7650664] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:25:58 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650664/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650664/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c91dacb660a04066b2743d358b3b6a05 Command failed (workunit test cephtool/test.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5074d4516d566e9d8b6aec912f26afd099de101 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7650669] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:06:58 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650669/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650669/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec8095bfa24f4b1d914d7cf3b4dfc6be Command failed on smithi087 with status 1: 'sudo yum -y install ceph- radosgw' [7650687] rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 02:06:10 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650687/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650687/ "2024-04-10T15:52:17.576203+0000 mon.a (mon.0) 396 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7650700] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:14:48 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650700/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650700/ "2024-04-10T15:47:39.072303+0000 mon.a (mon.0) 536 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7650712] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read} ----------------------------------------------------------------- time: 00:15:18 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650712/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650712/ sentry: https://sentry.ceph.com/organizations/ceph/?query=656e52bab27f4941944e7b04730f24dc Command failed on smithi055 with status 1: 'sudo yum -y install rbd-mirror' [7650713] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} ----------------------------------------------------------------- time: 00:10:19 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650713/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650713/ sentry: https://sentry.ceph.com/organizations/ceph/?query=104cae454b6d48538913a126772f0449 Command failed on smithi016 with status 1: 'sudo yum -y install ceph' [7650719] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650719/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650719/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42c03d7c48514790bdabaf65ac11b72e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650721] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650721/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650721/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4701f1f70a546d9af27f860ccf42954 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650723] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650723/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650723/ sentry: https://sentry.ceph.com/organizations/ceph/?query=371acb280e204d79ad3b1a3c1328a443 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650728] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650728/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650728/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6b4acab0f654e36bfc126aeb41e61ed HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650730] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650730/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650730/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0fbfb98b36f4ad1b9912648300dd39c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650732] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650732/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650732/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba7e722bfa604f4cae18c3363bb5a701 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650734] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{ubuntu_latest} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650734/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650734/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fe69cd7ab62449c89d4cc0992b29eca HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650736] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650736/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650736/ sentry: https://sentry.ceph.com/organizations/ceph/?query=359d26ff49d94b99a35bac51135cf4a1 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650738] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650738/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650738/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57f2fd0d4dc149168b0dd62609a9bf84 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650740] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650740/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650740/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca44e35c0b624ba9a5be873c1d810432 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650742] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650742/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650742/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67c9eda452414bfe9a80528c52d25166 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650743] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650743/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650743/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64ae559a8ebd4431ad52f3ba7f0f9f68 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650744] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650744/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650744/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c992438aed0d4099b5072ca36ce7c282 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650746] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650746/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650746/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9501577f6b204d71a355b2e95d229251 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650747] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650747/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650747/ sentry: https://sentry.ceph.com/organizations/ceph/?query=debc6417476c48f3b6e606a116085d5b HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650748] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650748/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=862d29ce50874d85ad1d3015037a82b9 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650749] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650749/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650749/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e8650457da84388af90e5da738db02e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650750] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650750/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650750/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70779098aca149d7ad4ca0824d2895bd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650751] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650751/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650751/ sentry: https://sentry.ceph.com/organizations/ceph/?query=14c4890005c74cfea9b8e0e0f9972e6c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650752] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650752/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650752/ sentry: https://sentry.ceph.com/organizations/ceph/?query=995fb15b29014451a2b180b55ffdb503 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650753] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650753/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650753/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2023a3f51a264c259d3229b2c1d62b9f HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650754] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650754/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650754/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dff7572cdc204eb2a8a03a747ccd13bc HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650757] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650757/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650757/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71ae47722cee40b496d51e63963e9a0c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650758] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650758/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b6961141d3b44edf948608e737a759aa HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650759] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-redelete} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650759/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650759/ sentry: https://sentry.ceph.com/organizations/ceph/?query=898f65ec46034a9eb8d27599edccfc72 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650760] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650760/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650760/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed6e0e96e5b241c1a45bfaca3008256d HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650761] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650761/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650761/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f32da74059074e988cf3eb6403bfdb53 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650762] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650762/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650762/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a443d0f0779e470bbb94a3916dc2ecaf HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650763] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650763/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650763/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49b9b8ca0bdf4d4a94915c479cb8d4d3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650764] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650764/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650764/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e0c7c2bbf6c4fae972d448f88960c8e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650765] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650765/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650765/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ccfba6d6d8841539a6730c14447a8ac HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650766] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_latest} tasks/insights} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650766/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650766/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ec93785598246a381c3cd84ee71548c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650767] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650767/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650767/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89f6e1da47e84a8ca5206cc80d81d62e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650768] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650768/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650768/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b55979853f4849eea0ccebbae436d2ed HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650769] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650769/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650769/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1a3f9b910ff4fef93c6d363cf8902dd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650770] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650770/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650770/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c835bb39b7b5407b81c685647cc5eaee HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650772] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650772/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650772/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fa801f063634f0eac5a8563b00d9dd0 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650773] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650773/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650773/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f960fb85ccd48cea4a1ceb05956a49f HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650774] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650774/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650774/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2be8aa882bbe4a658e97bd2227acc482 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650775] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650775/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650775/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d94f60ab61e4cf68a2872f52b6d3180 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650776] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650776/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650776/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae3b9a3550734bf6ab63e20fc1b60545 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650777] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650777/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650777/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ef15f19b68e41dba8b4ae1c9d2a9c8d HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650778] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650778/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650778/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1cc4399336b4e4b9ed4b4acaf836f38 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650779] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650779/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=283b2e614ade495a9d16c490aa7d4b60 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650780] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650780/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca68c9304282449680b63a6154b3351e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650782] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650782/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650782/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1430ab82d4484ef5b7753013dc8daa4d HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650783] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650783/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9439d3d5186a413b8fc7eac500ad5037 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650784] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650784/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650784/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06680767f6084fb6a2ee064161797be5 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650785] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-hybrid} tasks/e2e} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650785/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650785/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a67df1495419496f9735fe84cc919259 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650786] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650786/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650786/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b51ecb71256e468a8d65992790a24640 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650787] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650787/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77e507acc7154efe96c8a4d4fc24f36b HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650788] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650788/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650788/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76f4988855c34a5780294a19efece1f1 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650789] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650789/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650789/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fca96d1133394827b27101c22079edea HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650790] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650790/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650790/ sentry: https://sentry.ceph.com/organizations/ceph/?query=948a93209bfc42cba883a0ba7d68adba HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650791] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650791/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c58b28b45f3f437aa1e4f501821a1f4a HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650792] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650792/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650792/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c42b01cc080440d1824691c56dbc65ad HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650793] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650793/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4a35004538d843508134fb230d2527dd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650794] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650794/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650794/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f71fc922d72d481ab17815d6441b129c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650795] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650795/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650795/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a6083f844eb64b30823cbaf91f25bbbd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650796] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650796/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650796/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b7b52f98a95d44fda5babbd27a876da2 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650797] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650797/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650797/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b756593435b42608797d284366fa2d9 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650798] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650798/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650798/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a726e46b66c742c7b04e1c1fdea71761 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650799] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mgr} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650799/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650799/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46e5662140ab46449733c3d5c7aa5b83 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650800] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650800/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650800/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35df6f77ddfc4f95a8b7c47458ccc788 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650801] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650801/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650801/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c8633fda66945fb926efbba7a8a686b HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650802] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650802/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650802/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ddfd9d31b062447c9af2da9155a7cc02 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650803] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650803/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650803/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6167959f26474b4ab6921b4d18d07a92 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650804] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650804/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650804/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4f522c7840a452385d3ea866a0d3fd3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650805] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650805/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650805/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0112cba36084a6c96566fdbfe3dfdfc HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650806] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650806/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f344b38af68044548eb8e8ef82e7f170 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650807] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650807/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650807/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da2faed8fa8343f292ced019ed4b04f4 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650808] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650808/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650808/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e4538a86a19420d8fba9f52f90d049e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650809] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650809/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650809/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9bc7e96449a4acc898bd214e9fb082c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650810] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650810/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0b3bbceb56d4837857d21d7992ca0c1 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650811] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650811/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650811/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd4d6576368d4710aee5428074a50027 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650812] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650812/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650812/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64aa308634784f588ed670055241e027 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650813] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650813/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650813/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6913f54ad384219bbe381bd37e1c4eb HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650814] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_big} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650814/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650814/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9cf7bc06d0d54e769710c36ceaf765ef HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650815] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650815/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650815/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b9c783125702419ea5f43b001b5ace1a HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650816] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650816/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650816/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bddff0249228417aaf26b4da1daa7b34 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650817] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650817/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650817/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ccbe281ef8ec4b7ab7da536aad4ed703 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650819] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650819/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b6dc445f60094c51a6e0f625fd65b024 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650820] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650820/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650820/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d5e748cfc184042b6f025032541ad77 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650821] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650821/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650821/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db6a6f2293714bd9af50d4b030840e78 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650822] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650822/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650822/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd3215269f174050b4cbdc65af6efe3e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650823] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650823/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650823/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb4cab4bc5a64e04bc9c39a32fd8430d HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650824] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650824/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73b0d31ba3424ea58f14827dfe5b43d9 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650825] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/dedup-io-mixed} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650825/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650825/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc96bbdbd1a947c09d1ec223692ac607 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650826] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650826/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650826/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b4edbf0015b847bea89b0b279ee67160 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650827] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650827/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650827/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27f3b69975f440b5887663d35ba00b99 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650828] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650828/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650828/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a01c22c924e049239c009a04284fe2c8 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650829] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650829/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650829/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3472b28c6f654fd9ac28594376cf35ac HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650830] rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650830/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650830/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d2e8c944a17465eafa3a3a6f8aead6f HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650832] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650832/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650832/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64fc593eebfa44538c5ef0ad9c4b692c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650833] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650833/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650833/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4834f7388861411c83c09c5ccf591006 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650834] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects-redelete} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650834/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650834/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a77f1acd3a654bf1843b530634f5f924 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650835] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650835/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650835/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38f4701fa5c94d8b906133f85a598ecd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650836] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mix} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650836/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650836/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f0235bf51c043f6b89234b4eda1270f HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650837] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650837/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650837/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1f1d047233844e982feb37b674cf1c4 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650838] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650838/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650838/ sentry: https://sentry.ceph.com/organizations/ceph/?query=17bc0dd7776949c380ef6c1b0aea56ad HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650839] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650839/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650839/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08b36a32edfa41e883dd86caf27401f4 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650840] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650840/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650840/ sentry: https://sentry.ceph.com/organizations/ceph/?query=167521a412164265a4bcaf97db92b2a0 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650841] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650841/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e364125008e04ce1847a6daf10ed4f11 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650842] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650842/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650842/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2395a0c321ae4420be13fd17cfbf87e1 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650843] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650843/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650843/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f416ef2f4804d12b9c910d458a536cb HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650844] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650844/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650844/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc73456b0a5a4a71b08fefbbfcaca58a HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650845] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650845/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650845/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62129913f788449683649f7175a3c2c8 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650846] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650846/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650846/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c67360f254d74a89b491c255b72fcd38 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650847] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650847/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650847/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eafe16ddcf1c474a8d0d7cf20fccc39f HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650848] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650848/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650848/ sentry: https://sentry.ceph.com/organizations/ceph/?query=487b79f9258a47f4b83c6e74c8b5fe55 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650849] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650849/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=676d4def90d041379875cf8338aaffdd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650850] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/progress} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650850/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650850/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e324a06a0372411b8bdb9546094df365 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650851] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650851/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650851/ sentry: https://sentry.ceph.com/organizations/ceph/?query=334c0da627c7427083dbcd4e61dd7ac2 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650852] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650852/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650852/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e4793bd7d8d4308a0e6d32b9e064215 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650853] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650853/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650853/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d66a0a3068a24d5eb489db30a4f1da96 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650854] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650854/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7df3630756454a36ab70d1f6e82d5f95 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650855] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650855/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650855/ sentry: https://sentry.ceph.com/organizations/ceph/?query=077ce85a205a452193792ce5d15aed48 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650857] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650857/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650857/ sentry: https://sentry.ceph.com/organizations/ceph/?query=beee31de4d6a42a491ee659ad3ca086a HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650858] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650858/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650858/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6db1ee42b3754c2d9bb633313df8cd15 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650859] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650859/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2bcc40be7c704975a7c76371dc09f17d HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650860] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650860/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650860/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f91e29c015f8403681468aba32a39800 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650861] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/dashboard} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650861/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650861/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31dde2d8236b488190cf21032a1bddd4 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650862] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650862/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650862/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c7bdacbec52e4dbdbc35010f7008d9d4 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650863] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650863/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650863/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8804a7797809448998c5360d293b7772 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650864] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650864/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45605dc67de14154ac31424826bc147f HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650865] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650865/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0f520c83a534ce496d09f037e5fb9c3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650866] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650866/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650866/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e70a4c2832224a3bbaf3a054773219ca HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650867] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650867/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650867/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b2ac79163edb47c59545c6c6aee3a973 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650869] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650869/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650869/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca8cb6afc2cb4e6594b938f52df04840 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650871] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650871/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650871/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9917fa4a3194ea7bceaaf9195c4a4ca HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650873] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650873/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650873/ sentry: https://sentry.ceph.com/organizations/ceph/?query=44aee6a75e0f4700ae58151fd0c760ea HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650875] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650875/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650875/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2bfe0c452def47ce95a09135985e45b6 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650877] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650877/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650877/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abe287604bf645e5a93f07a59cf36b0a HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650879] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650879/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650879/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7379763090c94fcd929fe953b77cf7a5 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650881] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650881/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650881/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78439bdeb05b4781b8b3a3ec57fc38b0 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650884] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650884/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650884/ sentry: https://sentry.ceph.com/organizations/ceph/?query=628f8e5be9e84ecb9a25e610caa28392 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650885] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650885/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650885/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffd489c1f2ee4b3b9fc48b6b775af3ae HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650887] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650887/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650887/ sentry: https://sentry.ceph.com/organizations/ceph/?query=90b96e4c985243d4a0c0cd6a54b8f5cd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650889] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650889/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650889/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7bfe3e59c4142baa80648b599d74164 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650891] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/prometheus} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650891/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650891/ sentry: https://sentry.ceph.com/organizations/ceph/?query=10e0bef072f54b3db484b2e5aa9a2710 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650893] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650893/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650893/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54647c1c3a274eeb88ca245187874fc0 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650895] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650895/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650895/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d84fec051bb44e6d8b77525ae80229a6 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650897] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650897/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650897/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c2d260498564e5288eeaf45401b5aff HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650899] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650899/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=685e8e2438e64d3689ed6a2d291537fe HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650901] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650901/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650901/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc846c0568c04179b25ffcb1979f200b HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650903] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650903/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650903/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a5701e4683b4ba6aaf531109cb2bc7a HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650905] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650905/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650905/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fcff7643cca3453c8b3867761148c6a4 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650907] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650907/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650907/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8c0a260d26a4502b51ea6258d5d6ba4 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650909] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650909/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650909/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78842d94094847288a4d7bc2e80c2d7e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650911] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650911/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650911/ sentry: https://sentry.ceph.com/organizations/ceph/?query=97a476526169473d84cb858c2da18cdd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650913] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650913/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650913/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ee1e757fbbd4d5398e7cdfba9aec55b HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650915] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650915/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650915/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f50b946b8eb24fab8f09058c21d61c2c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650916] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650916/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650916/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4a9bacdf698e4a47bb89f42b09a02c24 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650918] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650918/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650918/ sentry: https://sentry.ceph.com/organizations/ceph/?query=750bedae4f3e48c4a06cc986d0487e58 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650920] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650920/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650920/ sentry: https://sentry.ceph.com/organizations/ceph/?query=472937140228442193218347aea198bc HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650924] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650924/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f4a080edc0c4cfab8394d7603c8d404 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650926] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650926/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cf5bff4427a94c68af7bf65a4ea666bf HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650928] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650928/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650928/ sentry: https://sentry.ceph.com/organizations/ceph/?query=899535fba0614df08814ef7eb3a3aab1 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650930] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650930/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650930/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8b8285ccb5d4f0a978eaec16456f82d HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650932] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650932/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650932/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1a2544f280e43ff9ae08c1a7dc61810 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650934] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650934/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650934/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07844f071a3e483593f4bfaa22bc8f33 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650936] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650936/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650936/ sentry: https://sentry.ceph.com/organizations/ceph/?query=108b4087a69245a9a26727e050d7d06d HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650939] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650939/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650939/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f27b84588b0543a1bd721bc687ac0a48 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650941] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/repair_test} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650941/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03764c75bda64098a9387c396a7cd577 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650943] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650943/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650943/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93c599405a754f4d941d3bad243bba33 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650945] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650945/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650945/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e08f141524824035b0d723588d2c52c0 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650947] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_latest} tasks/workunits} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650947/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650947/ sentry: https://sentry.ceph.com/organizations/ceph/?query=792ba155baa744b1982d2e77dec7573f HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650949] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650949/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650949/ sentry: https://sentry.ceph.com/organizations/ceph/?query=99cd88c1cf934edb8273ea69b15d11d1 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650951] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650951/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650951/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0dfa06d63ef342d4ac47ca61e16a7337 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650953] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650953/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650953/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c447ff272d84b499ec0146b8d210f37 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650955] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650955/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650955/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8df92df54f684c4aabd2b82186b92ba3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650956] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650956/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650956/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c6b4ac400d8417192326e17cb36e97c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650958] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650958/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650958/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8ba21ccc2224817a9a413d737c0426a HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650959] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650959/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650959/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db1cf46888114590bb54edffcb2160eb HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650961] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650961/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650961/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f7d620a995f42f4a4307fd9d44bd614 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650963] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650963/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650963/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2915cebadd94506a61d24d5ca767bb5 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650965] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650965/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650965/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88b2860e14534e89ae8e145362f6e5a0 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650967] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650967/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650967/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87780c0e4f974ed78f2cc33a2fa5dfbb HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650969] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650969/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650969/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8bace1de8eb84997b06186f9afed0dc7 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650971] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-redelete} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650971/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650971/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5d3deb4d7084dd5b990adebe9c6f66c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650974] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650974/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650974/ sentry: https://sentry.ceph.com/organizations/ceph/?query=910f10363c364208bf8c0d01a5623441 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650976] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650976/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650976/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dfd41be1a6ab4e799c98699caf90bc95 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650978] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650978/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650978/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c64a0e904a8e4a1e906117d311043214 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650980] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650980/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650980/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a51c0bf163cd44ebb0c36755edc85ce3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650982] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650982/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650982/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2829d863f764f0d85f137a0ec518f26 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650983] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650983/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650983/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34f43056e8f14929bf17f4dd954be026 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650985] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650985/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650985/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bedd9ab71c5d48228b2a2e012dcb3a88 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650986] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650986/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650986/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52306dbccbb74e57a66824b44a5d7bfc HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650988] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650988/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650988/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c4577c412f44aeea0b0bf419c196a51 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650990] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650990/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43070c8fd7024edf80706b1025ab911b HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650994] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650994/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650994/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0b934fc30ed4f35a6fdeb0da3c26378 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650996] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650996/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650996/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fcdd16c3fcf64cef9fa74f9b0a2438bd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7650998] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650998/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650998/ sentry: https://sentry.ceph.com/organizations/ceph/?query=844949bc06254978b793c1e960dc45e1 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651000] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651000/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651000/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a98f910fe2064d41b6ae5dc0a08c2016 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651002] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-radosbench} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651002/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651002/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35ae0d1bc86d456b998ba890106f1071 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651005] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/e2e} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651005/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651005/ sentry: https://sentry.ceph.com/organizations/ceph/?query=086d07f37db0441fbc0e848157841625 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651007] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/crash} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651007/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651007/ sentry: https://sentry.ceph.com/organizations/ceph/?query=12fe6f3658e04b2ab7ee41354e0e3a25 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651009] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651009/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651009/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b5d23ac1c6fb44c69521ae39518e488f HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651012] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651012/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651012/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff4e79d48bf649c688ee5a69d04770cc HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651015] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651015/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651015/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1cbcd650340471ca49341a99fde68b3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651017] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651017/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651017/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7af49fbcf06947e2bef78d639b1bdac7 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651018] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651018/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651018/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcf4630af02b403aa036dbbcb8cfd724 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651019] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651019/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651019/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1690296110546aaabcbcccd27b6f302 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651021] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651021/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651021/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0ebb2c6757d470895b1fef6968729e3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651026] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651026/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651026/ sentry: https://sentry.ceph.com/organizations/ceph/?query=146b45115b2c4961b8d158405a05d616 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651028] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651028/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651028/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0796bf5d010741c8b1388dda162c32b8 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651030] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651030/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651030/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d90381f7985443ab9b5b168b6dbe76d HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651031] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651031/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651031/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46c18499186245ea87d7392b175279b3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651032] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651032/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651032/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc889c3f84c14fd7886bc9b3332375c9 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651034] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651034/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651034/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa46483284434ff88fe9e6df43700edd HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651035] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651035/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651035/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad0ed3fb3d5f436b963a41da11a1b780 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651037] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651037/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651037/ sentry: https://sentry.ceph.com/organizations/ceph/?query=994c133cfa614b9e9e71f3b597a2a1da HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651039] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651039/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651039/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a6e16e2d5ff4ba9bce0183a8d1c4502 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651041] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651041/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651041/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9bcbee36de4b4139be91ec505fb3311c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651043] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651043/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651043/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df1c2d113a534ff6aeef2b38ee8186e6 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651046] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651046/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651046/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0b1476fce5104cec9a80c37cfd5668e3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651048] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651048/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651048/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5a0841a47f954d11a2bbbb18793b4682 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651050] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651050/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651050/ sentry: https://sentry.ceph.com/organizations/ceph/?query=777656d8505340a3a6aef4c9d967141e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651052] rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651052/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651052/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7587699a061463cb8ba867126ee7397 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651054] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651054/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651054/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e286a4c4fa242a98d0fbdacea25b479 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651058] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651058/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651058/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01803d85fb3d4a94b9f29e1cb5455532 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651060] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651060/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651060/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da3e37e570454451ba2d489f6c2fc014 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651061] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651061/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651061/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8fecdc9091545049b3109b8be506c59 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651067] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651067/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651067/ sentry: https://sentry.ceph.com/organizations/ceph/?query=faf52b3f11894400a34e533fa7038861 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651072] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651072/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651072/ sentry: https://sentry.ceph.com/organizations/ceph/?query=575bce2cf6fa4d659d98759279902891 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651074] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651074/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651074/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85a2de97c75543f0bcc3e178f325f01c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651075] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651075/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651075/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45a5c8339a3b40e0b170be105be856e4 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651077] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651077/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651077/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5576dd6aa17d4a3c95da4e6b111acd30 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651081] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651081/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651081/ sentry: https://sentry.ceph.com/organizations/ceph/?query=166cd4d683c74458891972b175641315 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651082] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/dedup-io-mixed} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651082/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651082/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6df20f9534e4becac404d36c63358c5 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651084] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651084/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651084/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9fc1cc04c9f343b69d139a5c095dab44 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651086] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651086/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651086/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e237fb855c9042d8a57145540af8c99e HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651088] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651088/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651088/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61905e8a2cb945579ef81b64364a7260 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651089] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651089/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651089/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e4305d945c1437392f0a1cbba2e71e3 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7651091] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/insights} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651091/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651091/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e8ea86e4cb84707a973390388aead69 HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) Dead ================================================================= [7650670] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650670/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650670/ hit max job timeout [7650684] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650684/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650684/ hit max job timeout [7650715] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:09:03 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650715/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650715/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ea64a25cce24ec08739a413f53e0a53 ['Failed to download key at http://download.ceph.com/keys/autobuild.asc: Request failed: ', 'Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: '] [7650717] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:08:50 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650717/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650717/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6dad01e8ee8c49ceb27fa511587fec66 ['Failed to download key at http://download.ceph.com/keys/autobuild.asc: Request failed: ', 'Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: '] [7650725] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650725/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650725/ Error reimaging machines: Failed to power on smithi047 [7650745] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:18:13 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef1b0e06ecd84b0d991a6c5c4d554f26 [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.or g/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] [7650755] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650755/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650755/ Error reimaging machines: Expected smithi165's OS to be centos 9 but found ubuntu 22.04 [7650756] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650756/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650756/ Error reimaging machines: list index out of range [7650771] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650771/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650771/ Error reimaging machines: reached maximum tries (241) after waiting for 3600 seconds [7650781] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:18:18 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781/ sentry: https://sentry.ceph.com/organizations/ceph/?query=86efc5142eb4401492c00391f43ea0bc [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.or g/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] [7650818] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:18:34 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e289e7eef024abdbd4a78824af59184 [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.or g/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] [7650831] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650831/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650831/ Error reimaging machines: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/174/task [7650856] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:18:20 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650856/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650856/ sentry: https://sentry.ceph.com/organizations/ceph/?query=818851c9bd934f1cb8e400d974b7888e [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.or g/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] [7650922] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:16:36 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650922/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650922/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e96e7321a774de6abbf6ced995189d5 [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.or g/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] [7650992] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:18:11 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650992/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650992/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5a471e7696f540e0a9578bde97447caf [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.or g/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] [7651011] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651011/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651011/ Error reimaging machines: Expected smithi008's OS to be ubuntu 22.04 but found centos 9 [7651023] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651023/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651023/ Error reimaging machines: reached maximum tries (241) after waiting for 3600 seconds [7651055] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:18:08 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055/ sentry: https://sentry.ceph.com/organizations/ceph/?query=537570e869f84fdfad8514d55867f48f [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.or g/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] [7651063] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651063/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651063/ Error reimaging machines: Expected smithi067's OS to be ubuntu 22.04 but found centos 9 [7651065] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651065/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651065/ Error reimaging machines: Expected smithi135's OS to be centos 9 but found ubuntu 22.04 [7651070] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651070/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651070/ Error reimaging machines: list index out of range [7651078] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651078/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651078/ Error reimaging machines: list index out of range Pass ================================================================= [7650642] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:44 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650642/ [7650644] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:29 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650644/ [7650648] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:24:13 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650648/ [7650650] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-balanced} time: 00:24:32 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650650/ [7650652] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench} time: 01:31:27 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650652/ [7650654] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:32:42 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650654/ [7650656] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:56 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650656/ [7650658] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:29:30 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650658/ [7650660] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:12 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650660/ [7650662] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect} time: 00:17:21 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650662/ [7650666] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:29 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650666/ [7650668] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:27:22 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650668/ [7650672] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} time: 00:23:36 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650672/ [7650674] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:13:21 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650674/ [7650676] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-bitmap} tasks/dashboard} time: 01:10:44 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650676/ [7650678] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/crash} time: 00:10:35 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650678/ [7650680] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_latest}} time: 00:35:14 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650680/ [7650682] rados/rest/{mgr-restful supported-random-distro$/{centos_latest}} time: 00:10:54 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650682/ [7650689] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:18:36 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650689/ [7650691] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:13:04 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650691/ [7650693] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:36 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650693/ [7650695] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:22 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650695/ [7650697] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:12:51 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650697/ [7650698] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:57 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650698/ [7650702] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:14:40 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650702/ [7650705] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} time: 00:32:33 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650705/ [7650706] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:11:58 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650706/ [7650708] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:18:16 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650708/ [7650710] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:22:45 info: https://pulpito.ceph.com/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650710/