2024-04-14T21:05:01.223 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-14T21:05:01.223 INFO:teuthology.results:subset: '25083/100000' 2024-04-14T21:05:01.223 INFO:teuthology.results:seed: '5975' 2024-04-15T02:07:52.995 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-15T02:07:53.019 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-15T14:08:02.424 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2024-04-15T14:08:26.268 INFO:scrape:Found 300 jobs 2024-04-15T14:08:33.393 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656308/teuthology.log 2024-04-15T14:08:43.923 INFO:scrape:Found 16 distinct failure reasons 2024-04-15T14:08:43.924 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-04-15T14:08:43.924 INFO:scrape:2 jobs: ['7656181', '7656328'] 2024-04-15T14:08:43.924 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-15T14:08:43.924 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-15T14:08:43.925 INFO:scrape: 2024-04-15T14:08:43.925 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2024-04-15T14:08:43.925 INFO:scrape:4 jobs: ['7656269', '7656344', '7656128', '7656200'] 2024-04-15T14:08:43.925 INFO:scrape:suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] 2024-04-15T14:08:43.925 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] 2024-04-15T14:08:43.925 INFO:scrape: 2024-04-15T14:08:43.925 INFO:scrape:Failure: Failed to reconnect to smithi016 2024-04-15T14:08:43.925 INFO:scrape:2 jobs: ['7656220', '7656149'] 2024-04-15T14:08:43.925 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados'] 2024-04-15T14:08:43.926 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/verify/{centos_latest', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'tasks/rados_api_tests', 'thrashers/mapgap', 'thrashosds-health', 'thrashosds-health}', 'validater/valgrind}', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] 2024-04-15T14:08:43.926 INFO:scrape: 2024-04-15T14:08:43.926 INFO:scrape:Failure: Command failed on smithi016 with status 1: 'sudo yum install -y kernel' 2024-04-15T14:08:43.926 INFO:scrape:4 jobs: ['7656221', '7656260', '7656315', '7656150'] 2024-04-15T14:08:43.926 INFO:scrape:suites intersection: [] 2024-04-15T14:08:43.926 INFO:scrape:suites union: ['agent/on', 'clusters/{fixed-4', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/objectstore/{backends/objectcacher-stress', 'rados/singleton-nomsgr/{all/health-warnings', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'task/test_cephadm_repos}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2024-04-15T14:08:43.926 INFO:scrape: 2024-04-15T14:08:43.926 INFO:scrape:Failure: Command failed on smithi111 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' 2024-04-15T14:08:43.926 INFO:scrape:52 jobs 2024-04-15T14:08:43.927 INFO:scrape:suites intersection: [] 2024-04-15T14:08:43.927 INFO:scrape:suites union: ['0-nvme-loop', '0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-install/octopus', '1-install/pacific', '1-install/quincy', '1-install/reef', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '2-services/rgw', '3-final}', 'agent/off', 'agent/on', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'debug/mgr', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc}', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream}', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04}', 'rados/cephadm/smoke/{0-distro/centos_9.stream', 'rados/cephadm/smoke/{0-distro/centos_9.stream_runc', 'rados/cephadm/smoke/{0-distro/ubuntu_22.04', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/dashboard/{0-single-container-host', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'random-objectstore$/{bluestore-comp-zlib}', 'random-objectstore$/{bluestore-hybrid}', 'start}', 'task/test_ca_signed_key}', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_iscsi_container/{centos_9.stream', 'task/test_monitoring_stack_basic}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/e2e}', 'test_iscsi_container}}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] 2024-04-15T14:08:43.927 INFO:scrape: 2024-04-15T14:08:43.927 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi088 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-04-15T14:08:43.927 INFO:scrape:3 jobs: ['7656119', '7656369', '7656233'] 2024-04-15T14:08:43.928 INFO:scrape:suites intersection: ['task/test_cephadm}'] 2024-04-15T14:08:43.928 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] 2024-04-15T14:08:43.928 INFO:scrape: 2024-04-15T14:08:43.928 INFO:scrape:Dead: b'2024-04-15T14:08:28.157 INFO:journalctl@ceph.mon.c.smithi088.stdout:Apr 15 14:08:27 smithi088 ceph-mon[38144]: pgmap v23725: 105 pgs: 105 active+clean; 4.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail' 2024-04-15T14:08:43.928 INFO:scrape:1 jobs: ['7656395'] 2024-04-15T14:08:43.928 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-04-15T14:08:43.928 INFO:scrape: 2024-04-15T14:08:43.928 INFO:scrape:Dead: b'2024-04-15T13:04:55.356 DEBUG:teuthology.exit:Finished running handlers' 2024-04-15T14:08:43.928 INFO:scrape:2 jobs: ['7656241', '7656243'] 2024-04-15T14:08:43.928 INFO:scrape:suites intersection: ['mon_election/connectivity'] 2024-04-15T14:08:43.928 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'supported-random-distro$/{centos_latest}}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-04-15T14:08:43.929 INFO:scrape: 2024-04-15T14:08:43.929 INFO:scrape:Dead: b'2024-04-15T02:23:04.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored' 2024-04-15T14:08:43.929 INFO:scrape:1 jobs: ['7656393'] 2024-04-15T14:08:43.929 INFO:scrape:suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-15T14:08:43.929 INFO:scrape: 2024-04-15T14:08:43.929 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-15T14:08:43.929 INFO:scrape:4 jobs: ['7656153', '7656286', '7656353', '7656192'] 2024-04-15T14:08:43.929 INFO:scrape:suites intersection: [] 2024-04-15T14:08:43.929 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}'] 2024-04-15T14:08:43.929 INFO:scrape: 2024-04-15T14:08:43.930 INFO:scrape:Failure: failed to install new kernel version within timeout 2024-04-15T14:08:43.930 INFO:scrape:1 jobs: ['7656190'] 2024-04-15T14:08:43.930 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-small-objects-many-deletes}'] 2024-04-15T14:08:43.930 INFO:scrape: 2024-04-15T14:08:43.930 INFO:scrape:Dead: None 2024-04-15T14:08:43.930 INFO:scrape:4 jobs: ['7656308', '7656307', '7656258', '7656189'] 2024-04-15T14:08:43.930 INFO:scrape: 2024-04-15T14:08:43.930 INFO:scrape:Failure: Command failed (workunit test mon/osd-erasure-code-profile.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/osd-erasure-code-profile.sh' 2024-04-15T14:08:43.930 INFO:scrape:1 jobs: ['7656272'] 2024-04-15T14:08:43.930 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] 2024-04-15T14:08:43.930 INFO:scrape: 2024-04-15T14:08:43.931 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-04-15T14:08:43.931 INFO:scrape:1 jobs: ['7656362'] 2024-04-15T14:08:43.931 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] 2024-04-15T14:08:43.931 INFO:scrape: 2024-04-15T14:08:43.931 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-04-15T14:08:43.931 INFO:scrape:1 jobs: ['7656129'] 2024-04-15T14:08:43.931 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-04-15T14:08:43.931 INFO:scrape: 2024-04-15T14:08:43.931 INFO:scrape:Failure: SSH connection to smithi137 was lost: 'sudo apt-get update' 2024-04-15T14:08:43.931 INFO:scrape:2 jobs: ['7656206', '7656316'] 2024-04-15T14:08:43.931 INFO:scrape:suites intersection: [] 2024-04-15T14:08:43.932 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', 'cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] 2024-04-15T14:08:43.932 INFO:scrape: 2024-04-15T14:08:43.944 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/ failed: 72 dead: 11 running: 2 waiting: 0 queued: 0 passed: 215 Fail ================================================================= [7656100] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:04:37 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656100/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656100/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a97f261aa4147d9afc41eaa69d8c907 Command failed on smithi163 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656105] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:07:05 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656105/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=faa19f15a7194cb4a2c0cbbed89dd597 Command failed on smithi138 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656112] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:07:30 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656112/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656112/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8212ee5de10a46a98af64f9b4de87cc1 Command failed on smithi111 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656119] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:13:57 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656119/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656119/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dba79290f5984cceb0a1abd40125c1ce Command failed (workunit test cephadm/test_cephadm.sh) on smithi088 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7656123] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:04:38 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656123/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656123/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c50a5a71c1f43e9bb44777c2343bcd5 Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656128] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:25:49 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656128/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656128/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad832da9af1d4d1db59f7fca6dc20fd7 Command failed (workunit test cephtool/test.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7656129] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:09:01 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656129/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656129/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46d48ca2f7d240529637557a6156ce3c Command failed (workunit test cephadm/test_repos.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7656134] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:08:54 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656134/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656134/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76f17506b516437c91be4453abb67311 Command failed on smithi059 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656138] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:07:21 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656138/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656138/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b7b88c94b8a4c0896da2dbdde46ed8d Command failed on smithi083 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656144] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:07:36 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656144/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656144/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cbcf5a88cd934965874477b0dfd9d300 Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656148] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:07:47 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656148/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656148/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a09cc2fe4134781841e5009ebe10128 Command failed on smithi111 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656149] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656149/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656149/ sentry: https://sentry.ceph.com/organizations/ceph/?query=faed511a2d9f475ba2888e14ed1e52b2 Failed to reconnect to smithi174 [7656153] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:59:57 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656153/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656153/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7656155] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:07:50 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656155/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656155/ sentry: https://sentry.ceph.com/organizations/ceph/?query=231c0d28171840fba11f7edc98106e06 Command failed on smithi097 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656160] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:04:42 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656160/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656160/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20b0bced0fc94ff589836dbd7bfc66fa Command failed on smithi148 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656165] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:08:13 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656165/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656165/ sentry: https://sentry.ceph.com/organizations/ceph/?query=122249bed7234867a0fc4d4a5dcf9ac3 Command failed on smithi046 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656168] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-hybrid} tasks/e2e} ----------------------------------------------------------------- time: 00:07:07 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656168/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656168/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49a3361392c7421481d1501e6246d659 Command failed on smithi067 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656171] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:08:57 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656171/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656171/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15dbe71172e540629ea7145874758392 Command failed on smithi134 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656172] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:04:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656172/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656172/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08b139f7a3544e2fb45f82f88658d3f7 Command failed on smithi142 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656180] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:04:42 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656180/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656180/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ab5395c9165419a846596e55b25d939 Command failed on smithi016 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656181] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:09:56 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656181/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656181/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31eccf850e8c4ccea0ff21a8e9179ccd Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7656185] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:04:34 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656185/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656185/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c29722c824204ca1b286b435cac66b34 Command failed on smithi031 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656190] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656190/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656190/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77add51075234c81ac506709a73bd25b failed to install new kernel version within timeout [7656191] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:07:45 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656191/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656191/ sentry: https://sentry.ceph.com/organizations/ceph/?query=832c2f22c28349cdb534f57a20c07f97 Command failed on smithi038 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656192] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:21:31 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656192/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656192/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7656197] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:07:54 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656197/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656197/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d99bf80ca7b24f46b62b839a49b1ebf8 Command failed on smithi005 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656200] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:27:09 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656200/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656200/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5526c3bda95b43919e4629015a6b250d Command failed (workunit test cephtool/test.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7656204] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:07:21 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656204/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aee1c880b17f496f861244d7361ad0a3 Command failed on smithi023 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656209] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:09:16 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656209/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7aac87cd7a0140db8995061c7563a352 Command failed on smithi044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656210] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:09:12 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656210/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656210/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f50ead91b893486fb135fe4e4ce0cbc0 Command failed on smithi052 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656216] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:04:36 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656216/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656216/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aff44bbab1c043eaac48d9e27257d8d3 Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656220] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656220/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656220/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03cb8146f1024fc6bd0f5fb0f976cf2a Failed to reconnect to smithi016 [7656221] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656221/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656221/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68eca55c056c4d78959be5c383807d78 Command failed on smithi016 with status 1: 'sudo yum install -y kernel' [7656224] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:04:42 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656224/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656224/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2baeafa5d40b40169bfa69a4c5399831 Command failed on smithi121 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656229] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:09:04 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656229/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656229/ sentry: https://sentry.ceph.com/organizations/ceph/?query=91e7e53e38d04a719ad43bdf29556ddb Command failed on smithi100 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656233] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:13:57 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656233/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656233/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1dd690afeca34383b37cc0cf60fc0b9a Command failed (workunit test cephadm/test_cephadm.sh) on smithi046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7656238] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:04:39 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656238/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656238/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6940814a6924e37abcab1991f86f8dd Command failed on smithi171 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656246] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:09:26 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656246/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656246/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd6b80960b8e45e195723366f4f8c9c4 Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656247] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656247/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656247/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a731108dcaf1439188949bb4beea1b49 Command failed on smithi050 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656255] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:04:40 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656255/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656255/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1931edef5a244fbbe2009a5a3a58f17 Command failed on smithi203 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656266] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:07:21 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656266/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656266/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b1df74430504aa09fd179d26f3fe627 Command failed on smithi088 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656269] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:26:03 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656269/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656269/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f4d965b9bc84c5a93e6ea29729bc1f6 Command failed (workunit test cephtool/test.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7656270] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:09:08 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656270/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656270/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c5d9130a60f42e3b9b777ca1c7161c1 Command failed on smithi129 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656272] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} ----------------------------------------------------------------- time: 00:46:34 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656272/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656272/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8e67a579ab44b54ae2a55637406a3ae Command failed (workunit test mon/osd-erasure-code-profile.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/osd-erasure-code- profile.sh' [7656277] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:04:34 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656277/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656277/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6806c0af3424ff898d3bad55ab6bca9 Command failed on smithi105 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656284] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:07:35 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656284/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656284/ sentry: https://sentry.ceph.com/organizations/ceph/?query=057c53d26d39400b841d6742f750f25f Command failed on smithi073 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656285] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:09:14 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656285/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656285/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00dafc67b972455abb31b04576ff278e Command failed on smithi064 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656286] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:29:16 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656286/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656286/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7656292] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:04:34 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656292/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656292/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d67c7fa12429479d99487890063ae58d Command failed on smithi080 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656296] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:07:02 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656296/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656296/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0832d427e1e0405282e903412797d3ca Command failed on smithi165 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656301] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:04:38 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656301/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656301/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3215064f6414e279b5081b426ab8eb7 Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656310] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:09:18 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656310/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656310/ sentry: https://sentry.ceph.com/organizations/ceph/?query=083b9b74b9eb490f8279424e91c39c9e Command failed on smithi045 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656317] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e} ----------------------------------------------------------------- time: 00:07:29 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656317/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656317/ sentry: https://sentry.ceph.com/organizations/ceph/?query=392e359c025c4748b7ba4d6698aa6898 Command failed on smithi118 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656321] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:07:20 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656321/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656321/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7769ef6569bf48abb7b77bfa381025cc Command failed on smithi047 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656323] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:09:25 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656323/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656323/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff4b277adaaa477a8cf85e2b7481d101 Command failed on smithi023 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656326] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:04:45 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656326/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656326/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f90666f8c044c34ab5dbd2446f043b6 Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656328] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:09:55 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656328/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e19353ca52614f4eb76f31d4f235f4de Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7656332] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:04:32 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656332/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656332/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d7bb26ea59b44e3b7777e8df7cd0159 Command failed on smithi139 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656338] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:08:02 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656338/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656338/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7461246891054c68997466cd838fbab6 Command failed on smithi073 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656344] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:26:05 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656344/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656344/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c92252376334dffa9869ed3408351ab Command failed (workunit test cephtool/test.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7656348] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:04:42 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656348/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656348/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d53cb1f2bf1454caf4a0b6cce08b4bf Command failed on smithi038 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656352] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:07:18 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656352/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656352/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2832a91b734f4dc69eac69dbad1b5d11 Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656353] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:52:02 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656353/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656353/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7656356] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:04:50 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656356/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b90254ee90e0428994d7c23731205520 Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656359] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:09:06 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656359/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656359/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5533c98222e5450f9b3b9bf7c0061c52 Command failed on smithi016 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656362] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} ----------------------------------------------------------------- time: 01:33:25 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656362/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656362/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5877365b9644f07b771594e45e8ee1c Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- repair.sh' [7656365] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:07:26 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656365/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656365/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f457cb938be44216a9ffcf5a158ad876 Command failed on smithi063 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656369] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:17:13 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656369/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656369/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ca1c64af4e9457cb88e5ea55757acab Command failed (workunit test cephadm/test_cephadm.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7656381] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:07:26 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656381/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=249afaaf604446f2a2e6f257f2555867 Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656386] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:07:12 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656386/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656386/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b478a55d6b774e9da67b1acee1ae8af5 Command failed on smithi084 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656397] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:04:35 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656397/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31b11ead67244d79a655948d05813373 Command failed on smithi131 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' [7656399] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:08:58 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656399/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656399/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41916afdc2b649549e3fee9593b73a45 Command failed on smithi083 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 pull' Dead ================================================================= [7656150] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656150/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656150/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a08a3c8d97e34a68b76368a818514cfe SSH connection to smithi174 was lost: 'sudo yum install -y kernel' [7656189] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656189/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656189/ Error reimaging machines: list index out of range [7656206] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656206/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656206/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8943f0268a7c4dc2af7699c3494dacfd SSH connection to smithi137 was lost: 'sudo apt-get update' [7656241] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656241/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656241/ hit max job timeout [7656243] rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656243/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656243/ hit max job timeout [7656258] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656258/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656258/ Error reimaging machines: Expected smithi113's OS to be ubuntu 22.04 but found centos 9 [7656260] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656260/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656260/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db19ab2594304e969cb9fe96baf3c5f3 SSH connection to smithi113 was lost: 'sudo yum install -y kernel' [7656307] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656307/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656307/ Error reimaging machines: Expected smithi057's OS to be ubuntu 22.04 but found centos 9 [7656308] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_latest} tasks/progress} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656308/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656308/ Error reimaging machines: Expected smithi057's OS to be centos 9 but found ubuntu 22.04 [7656315] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656315/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656315/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff5b2d61f433438a9f30931d53ac7db3 SSH connection to smithi046 was lost: 'sudo yum install -y kernel' [7656316] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656316/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656316/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7221580ee9684789a932b42f8863125a SSH connection to smithi057 was lost: 'sudo apt-get update' Running ================================================================= [7656393] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656393/ [7656395] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656395/ Pass ================================================================= [7656101] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:16:51 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656101/ [7656102] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:19:11 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656102/ [7656103] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:03 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656103/ [7656104] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:22:49 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656104/ [7656106] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:16:17 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656106/ [7656107] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:08:54 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656107/ [7656108] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:39 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656108/ [7656109] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:23 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656109/ [7656110] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:40 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656110/ [7656111] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:12:58 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656111/ [7656113] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:11 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656113/ [7656114] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:01 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656114/ [7656115] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:08:20 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656115/ [7656116] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:25 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656116/ [7656117] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:51 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656117/ [7656118] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:27 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656118/ [7656120] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} time: 00:41:16 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656120/ [7656121] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:23:59 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656121/ [7656122] rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:19:01 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656122/ [7656124] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:13:53 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656124/ [7656125] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-balanced} time: 00:22:14 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656125/ [7656126] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:24:30 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656126/ [7656127] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:02 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656127/ [7656130] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:45:34 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656130/ [7656131] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:12:40 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656131/ [7656132] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:13 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656132/ [7656133] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:13:03 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656133/ [7656135] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:19:23 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656135/ [7656136] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:14:54 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656136/ [7656137] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_latest}} time: 00:08:27 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656137/ [7656139] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:10:44 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656139/ [7656140] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:12:06 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656140/ [7656141] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656141/ [7656142] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656142/ [7656143] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:51 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656143/ [7656145] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:12:54 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656145/ [7656146] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:52 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656146/ [7656147] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:27 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656147/ [7656151] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_cls_all} time: 00:16:16 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656151/ [7656152] rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} time: 01:21:42 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656152/ [7656154] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:11:48 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656154/ [7656156] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:35 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656156/ [7656157] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656157/ [7656158] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:20:07 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656158/ [7656159] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read} time: 00:20:31 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656159/ [7656161] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:15:05 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656161/ [7656162] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:28 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656162/ [7656163] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:29:39 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656163/ [7656164] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:25:51 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656164/ [7656166] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/crash} time: 00:10:33 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656166/ [7656167] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:25:56 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656167/ [7656169] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:45:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656169/ [7656170] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:36 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656170/ [7656173] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:16 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656173/ [7656174] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:27:17 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656174/ [7656175] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/rados_python} time: 00:14:43 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656175/ [7656176] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:06 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656176/ [7656177] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:09:56 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656177/ [7656178] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:21:08 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656178/ [7656179] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:15:02 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656179/ [7656182] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:56 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656182/ [7656183] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:08 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656183/ [7656184] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:17:19 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656184/ [7656186] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:27:47 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656186/ [7656187] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/lockdep} time: 00:11:01 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656187/ [7656188] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:43 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656188/ [7656193] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:33:41 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656193/ [7656194] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:29 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656194/ [7656195] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/failover} time: 00:13:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656195/ [7656196] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:13:01 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656196/ [7656198] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch} time: 00:15:40 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656198/ [7656199] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 01:10:28 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656199/ [7656201] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache} time: 00:11:38 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656201/ [7656202] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:08:12 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656202/ [7656203] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:22 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656203/ [7656205] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:08:07 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656205/ [7656207] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:19:36 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656207/ [7656208] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:14 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656208/ [7656211] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/dedup-io-mixed} time: 00:15:44 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656211/ [7656212] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:14 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656212/ [7656213] rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} time: 01:05:29 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656213/ [7656214] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:38 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656214/ [7656215] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:58 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656215/ [7656217] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:13:59 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656217/ [7656218] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:12:00 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656218/ [7656219] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:23 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656219/ [7656222] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:11:21 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656222/ [7656223] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} time: 00:15:56 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656223/ [7656225] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/insights} time: 00:11:54 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656225/ [7656226] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:20:04 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656226/ [7656227] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:13:16 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656227/ [7656228] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:11:05 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656228/ [7656230] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} time: 00:28:02 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656230/ [7656231] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:32:07 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656231/ [7656232] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:11:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656232/ [7656234] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:19 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656234/ [7656235] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:35 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656235/ [7656236] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:12:36 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656236/ [7656237] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:25 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656237/ [7656239] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:16:39 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656239/ [7656240] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/dashboard} time: 01:11:17 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656240/ [7656242] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} time: 00:27:28 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656242/ [7656244] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:41:06 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656244/ [7656245] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:33:25 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656245/ [7656248] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:20 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656248/ [7656249] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:42 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656249/ [7656250] rados/singleton/{all/mon-config mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:15:07 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656250/ [7656251] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:56 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656251/ [7656252] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:09 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656252/ [7656253] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{centos_latest} tasks/module_selftest} time: 00:24:31 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656253/ [7656254] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/lockdep} time: 00:17:03 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656254/ [7656256] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:26:55 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656256/ [7656257] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:14:42 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656257/ [7656259] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:11:49 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656259/ [7656261] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:23 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656261/ [7656262] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:31:48 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656262/ [7656263] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:25:39 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656263/ [7656264] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/snaps-few-objects} time: 00:31:12 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656264/ [7656265] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench} time: 01:07:57 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656265/ [7656267] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:36 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656267/ [7656268] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mix} time: 00:20:06 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656268/ [7656271] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:12:32 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656271/ [7656273] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:13 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656273/ [7656274] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:26:01 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656274/ [7656275] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:13 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656275/ [7656276] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/redirect} time: 00:12:45 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656276/ [7656278] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:11:52 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656278/ [7656279] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:10 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656279/ [7656280] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:54 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656280/ [7656281] rados/singleton/{all/peer mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:08:38 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656281/ [7656282] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:14 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656282/ [7656283] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:27 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656283/ [7656287] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:15:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656287/ [7656288] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:16:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656288/ [7656289] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:33:53 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656289/ [7656290] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:23:50 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656290/ [7656291] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:48 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656291/ [7656293] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:28:58 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656293/ [7656294] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:23:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656294/ [7656295] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:16:03 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656295/ [7656297] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:15:12 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656297/ [7656298] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:47 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656298/ [7656299] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/sync workloads/pool-create-delete} time: 00:17:07 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656299/ [7656300] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:34 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656300/ [7656302] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill} time: 02:25:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656302/ [7656303] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:08:54 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656303/ [7656304] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:32 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656304/ [7656305] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:08:48 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656305/ [7656306] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:14 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656306/ [7656309] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:14:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656309/ [7656311] rados/singleton/{all/radostool mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:14:30 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656311/ [7656312] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:05 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656312/ [7656313] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:14 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656313/ [7656314] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/readwrite} time: 00:12:11 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656314/ [7656318] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} time: 00:21:20 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656318/ [7656319] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:25:45 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656319/ [7656320] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/lockdep} time: 00:40:01 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656320/ [7656322] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:27 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656322/ [7656324] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656324/ [7656325] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:26:03 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656325/ [7656327] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:15:14 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656327/ [7656329] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:21:44 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656329/ [7656330] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_5925} time: 00:11:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656330/ [7656331] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} time: 02:31:26 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656331/ [7656333] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:23:53 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656333/ [7656334] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:18:39 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656334/ [7656335] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:18:33 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656335/ [7656336] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{centos_latest} tasks/prometheus} time: 00:11:29 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656336/ [7656337] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:37 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656337/ [7656339] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:16:49 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656339/ [7656340] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656340/ [7656341] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects} time: 00:21:17 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656341/ [7656342] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:05 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656342/ [7656343] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:11:10 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656343/ [7656345] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:15:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656345/ [7656346] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:15 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656346/ [7656347] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656347/ [7656349] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{ubuntu_latest}} time: 00:09:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656349/ [7656350] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:10:22 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656350/ [7656351] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:33 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656351/ [7656354] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:38 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656354/ [7656355] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 01:56:59 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656355/ [7656357] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/scrub_test} time: 00:10:13 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656357/ [7656358] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:11:17 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656358/ [7656360] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:18:40 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656360/ [7656361] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:54 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656361/ [7656363] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:36 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656363/ [7656364] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:10:53 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656364/ [7656366] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_api_tests} time: 00:43:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656366/ [7656367] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 02:03:13 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656367/ [7656368] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:57 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656368/ [7656370] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:25:25 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656370/ [7656371] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:25 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656371/ [7656372] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:28:05 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656372/ [7656373] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:45:15 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656373/ [7656374] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} time: 00:06:17 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656374/ [7656375] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:22:33 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656375/ [7656376] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:27:39 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656376/ [7656377] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:46 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656377/ [7656378] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:16 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656378/ [7656379] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:12:16 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656379/ [7656380] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:20:28 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656380/ [7656382] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:27 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656382/ [7656383] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:49 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656383/ [7656384] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:25 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656384/ [7656385] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/lockdep} time: 00:11:06 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656385/ [7656387] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:19:58 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656387/ [7656388] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:21:47 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656388/ [7656389] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_latest} tasks/crash} time: 00:10:45 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656389/ [7656390] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zstd} tasks/dashboard} time: 01:11:03 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656390/ [7656391] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:40:14 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656391/ [7656392] rados/rest/{mgr-restful supported-random-distro$/{centos_latest}} time: 00:13:02 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656392/ [7656394] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} time: 00:15:24 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656394/ [7656396] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:16:20 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656396/ [7656398] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests} time: 00:26:25 info: https://pulpito.ceph.com/teuthology-2024-04-14_21:00:09-rados-squid-distro-default-smithi/7656398/