2023-05-24T23:04:37.346 INFO:root:teuthology version: 0.0.1.dev109+ge5a3f9b 2023-05-24T23:04:37.347 INFO:teuthology.results:subset: '111/120000' 2023-05-24T23:04:37.347 INFO:teuthology.results:seed: '884' 2023-05-25T02:33:40.922 INFO:root:teuthology version: 0.0.1.dev109+ge5a3f9b 2023-05-25T02:33:40.977 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-25T05:40:03.102 INFO:teuthology.results:Tests finished! gathering results... 2023-05-25T05:40:26.957 INFO:scrape:Found 323 jobs 2023-05-25T05:40:27.220 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285910/teuthology.log 2023-05-25T05:40:33.653 INFO:scrape:Found 15 distinct failure reasons 2023-05-25T05:40:33.654 INFO:scrape:Failure: {'smithi107.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['rpm', '-e', 'ceph-debuginfo'], 'delta': '0:00:00.012461', 'end': '2023-05-25 02:11:42.006067', 'invocation': {'module_args': {'_raw_params': 'rpm -e ceph-debuginfo', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 1, 'start': '2023-05-25 02:11:41.993606', 'stderr': 'error: Failed dependencies:\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mds-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-cephfs-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-radosgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) libcephfs2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-mirror-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-base-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-test-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-osd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-nbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-common-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mgr-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librbd1-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-immutable-object-cache-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados-devel-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mon-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rados-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', 'stderr_lines': ['error: Failed dependencies:', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mds-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-cephfs-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-radosgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) libcephfs2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-mirror-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-base-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-test-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-osd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-nbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-common-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mgr-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librbd1-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-immutable-object-cache-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados-devel-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mon-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rados-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64'], 'stdout': '', 'stdout_lines': []}, 'smithi159.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} 2023-05-25T05:40:33.654 INFO:scrape:1 jobs: ['7285901'] 2023-05-25T05:40:33.655 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools'] 2023-05-25T05:40:33.655 INFO:scrape: 2023-05-25T05:40:33.655 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d9bb805bc69c581615cf909b058de5a74f7f4fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2023-05-25T05:40:33.656 INFO:scrape:2 jobs: ['7285672', '7285835'] 2023-05-25T05:40:33.656 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'tasks/e2e}'] 2023-05-25T05:40:33.656 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'tasks/e2e}'] 2023-05-25T05:40:33.657 INFO:scrape: 2023-05-25T05:40:33.657 INFO:scrape:Dead: None 2023-05-25T05:40:33.657 INFO:scrape:11 jobs: ['7285910', '7285952', '7285853', '7285897', '7285961', '7285900', '7285894', '7285933', '7285946', '7285896', '7285943'] 2023-05-25T05:40:33.658 INFO:scrape: 2023-05-25T05:40:33.658 INFO:scrape:Failure: machine smithi038.front.sepia.ceph.com is locked by scheduled_teuthology@teuthology, not scheduled_yuriw@teuthology 2023-05-25T05:40:33.658 INFO:scrape:1 jobs: ['7285859'] 2023-05-25T05:40:33.659 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}'] 2023-05-25T05:40:33.659 INFO:scrape: 2023-05-25T05:40:33.659 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_refcount.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh' 2023-05-25T05:40:33.660 INFO:scrape:1 jobs: ['7285758'] 2023-05-25T05:40:33.660 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_3.0}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-05-25T05:40:33.660 INFO:scrape: 2023-05-25T05:40:33.661 INFO:scrape:Failure: Command failed on smithi079 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" 2023-05-25T05:40:33.661 INFO:scrape:4 jobs: ['7285673', '7285755', '7285836', '7285916'] 2023-05-25T05:40:33.661 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '3-final', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04'] 2023-05-25T05:40:33.662 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}'] 2023-05-25T05:40:33.662 INFO:scrape: 2023-05-25T05:40:33.662 INFO:scrape:Failure: {'smithi100.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} 2023-05-25T05:40:33.663 INFO:scrape:7 jobs: ['7285899', '7285903', '7285898', '7285902', '7285947', '7285948', '7285895'] 2023-05-25T05:40:33.663 INFO:scrape:suites intersection: [] 2023-05-25T05:40:33.663 INFO:scrape:suites union: ['agent/off', 'clusters/{fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'openstack', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'rados/objectstore/{backends/fusestore', 'rados/perf/{ceph', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'rados/singleton/{all/divergent_priors2', 'scheduler/wpq_default_shards', 'settings/optimized', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'task/test_nfs}', 'tasks/rados_workunit_loadgen_mostlyread}', 'tasks/repair_test}', 'ubuntu_latest', 'workloads/sample_fio}'] 2023-05-25T05:40:33.664 INFO:scrape: 2023-05-25T05:40:33.664 INFO:scrape:Failure: Cannot connect to remote host smithi177 2023-05-25T05:40:33.664 INFO:scrape:2 jobs: ['7285958', '7285956'] 2023-05-25T05:40:33.665 INFO:scrape:suites intersection: [] 2023-05-25T05:40:33.665 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', 'cluster/{12-osds', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}'] 2023-05-25T05:40:33.665 INFO:scrape: 2023-05-25T05:40:33.665 INFO:scrape:Failure: Command failed on smithi023 with status 1: 'sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh collectl' 2023-05-25T05:40:33.666 INFO:scrape:5 jobs: ['7285945', '7285944', '7285907', '7285949', '7285955'] 2023-05-25T05:40:33.666 INFO:scrape:suites intersection: [] 2023-05-25T05:40:33.666 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{default}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'openstack', 'openstack}', 'rados', 'rados/perf/{ceph', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-partial-recovery}', 'scheduler/dmclock_default_shards', 'settings/optimized', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashosds-health', 'ubuntu_latest', 'workloads/dedup-io-snaps}', 'workloads/ec-small-objects-overwrites}', 'workloads/fio_4K_rand_read}', 'workloads/redirect}'] 2023-05-25T05:40:33.667 INFO:scrape: 2023-05-25T05:40:33.667 INFO:scrape:Failure: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) 2023-05-25T05:40:33.667 INFO:scrape:2 jobs: ['7285669', '7285807'] 2023-05-25T05:40:33.668 INFO:scrape:suites intersection: ['agent/off', 'mon_election/classic', 'task/test_nfs}'] 2023-05-25T05:40:33.668 INFO:scrape:suites union: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_nfs}'] 2023-05-25T05:40:33.668 INFO:scrape: 2023-05-25T05:40:33.669 INFO:scrape:Failure: {'smithi177.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi177.front.sepia.ceph.com port 22: No route to host'}} 2023-05-25T05:40:33.669 INFO:scrape:1 jobs: ['7285909'] 2023-05-25T05:40:33.669 INFO:scrape:suites: ['0-nvme-loop', 'agent/off', 'fixed-2', 'mon_election/classic', 'rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'start}'] 2023-05-25T05:40:33.670 INFO:scrape: 2023-05-25T05:40:33.670 INFO:scrape:Failure: Stale jobs detected, aborting. 2023-05-25T05:40:33.670 INFO:scrape:1 jobs: ['7285918'] 2023-05-25T05:40:33.670 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_rhel8}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-05-25T05:40:33.671 INFO:scrape: 2023-05-25T05:40:33.671 INFO:scrape:Failure: "2023-05-25T00:18:25.508837+0000 mon.a (mon.0) 2108 : cluster [WRN] Health check failed: 1 pools have too many placement groups (POOL_TOO_MANY_PGS)" in cluster log 2023-05-25T05:40:33.671 INFO:scrape:1 jobs: ['7285745'] 2023-05-25T05:40:33.672 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{rhel_8}}'] 2023-05-25T05:40:33.672 INFO:scrape: 2023-05-25T05:40:33.672 INFO:scrape:Failure: Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi022 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d9bb805bc69c581615cf909b058de5a74f7f4fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' 2023-05-25T05:40:33.673 INFO:scrape:1 jobs: ['7285819'] 2023-05-25T05:40:33.673 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb', 'supported-random-distro$/{rhel_8}}'] 2023-05-25T05:40:33.673 INFO:scrape: 2023-05-25T05:40:33.674 INFO:scrape:Failure: SSH connection to smithi162 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' 2023-05-25T05:40:33.674 INFO:scrape:1 jobs: ['7285908'] 2023-05-25T05:40:33.674 INFO:scrape:suites: ['mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/version-number-sanity', 'supported-random-distro$/{rhel_8}}'] 2023-05-25T05:40:33.674 INFO:scrape: 2023-05-25T05:40:33.686 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/ failed: 21 dead: 20 running: 0 waiting: 0 queued: 0 passed: 282 Fail ================================================================= [7285669] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:26:23 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285669/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285669/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f63e465712a4229ba4c112b6a1f7c3a Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) [7285672] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-lz4} tasks/e2e} ----------------------------------------------------------------- time: 00:19:34 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285672/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285672/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8195cf04dd64968acbcc4581bf13986 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d9bb805bc69c581615cf909b058de5a74f7f4fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7285673] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/master} ----------------------------------------------------------------- time: 00:07:16 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285673/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285673/ sentry: https://sentry.ceph.com/organizations/ceph/?query=737c466c9e414e46b033d8a4657acd64 Command failed on smithi079 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7285745] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:39:15 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285745/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285745/ "2023-05-25T00:18:25.508837+0000 mon.a (mon.0) 2108 : cluster [WRN] Health check failed: 1 pools have too many placement groups (POOL_TOO_MANY_PGS)" in cluster log [7285755] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} ----------------------------------------------------------------- time: 00:05:54 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285755/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285755/ sentry: https://sentry.ceph.com/organizations/ceph/?query=508e0d52292e4fa2aa4f2678125acc12 Command failed on smithi149 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7285758] rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:42:30 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285758/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d222d5a7e25643789420544370dea4af Command failed (workunit test cls/test_cls_refcount.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh' [7285807] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:27:08 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285807/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285807/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c1bce360112c442e803d4815619131de Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) [7285819] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:13:54 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285819/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1b35fb938a143de92c81d3e5e7a5f83 Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi022 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d9bb805bc69c581615cf909b058de5a74f7f4fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' [7285835] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} tasks/e2e} ----------------------------------------------------------------- time: 00:25:55 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285835/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285835/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa0cb95ff5ac4f51ad0de47c21692f65 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d9bb805bc69c581615cf909b058de5a74f7f4fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7285836] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} ----------------------------------------------------------------- time: 00:06:46 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285836/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285836/ sentry: https://sentry.ceph.com/organizations/ceph/?query=993c9647321845818a6d686596678d2d Command failed on smithi019 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7285859] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285859/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19fc5819b40e466382be17dbb46578e3 machine smithi038.front.sepia.ceph.com is locked by scheduled_teuthology@teuthology, not scheduled_yuriw@teuthology [7285907] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285907/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285907/ sentry: https://sentry.ceph.com/organizations/ceph/?query=661249147a02463fa6be917c6de0c44c Command failed on smithi162 with status 100: 'sudo apt-get clean' [7285909] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:07:15 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285909/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285909/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4dea17564658441c9b2098369a940af4 {'smithi177.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi177.front.sepia.ceph.com port 22: No route to host'}} [7285916] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/1.7.2} ----------------------------------------------------------------- time: 00:06:22 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285916/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285916/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8db2760519a6476ba4ec58093968a826 Command failed on smithi057 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed- by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils" [7285918] rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285918/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285918/ sentry: https://sentry.ceph.com/organizations/ceph/?query=30ac1679889e47dfba00d986778f6fc1 Stale jobs detected, aborting. [7285944] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285944/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285944/ sentry: https://sentry.ceph.com/organizations/ceph/?query=416b9fa9c3704bdc8effac26f152a802 Command failed on smithi177 with status 1: 'sudo yum install -y kernel' [7285945] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} ----------------------------------------------------------------- time: 00:13:34 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285945/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285945/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0b0cc1b87add40599b7326442c300abc Command failed on smithi023 with status 1: 'sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh collectl' [7285949] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285949/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285949/ sentry: https://sentry.ceph.com/organizations/ceph/?query=944e6bc6fc75418e872502e3169084fe Command failed on smithi162 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' [7285955] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285955/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285955/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b316e2febb8843809f7c6a5cc568db1c Command failed on smithi038 with status 1: 'sudo yum install -y kernel' [7285956] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285956/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285956/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e96ba40acd494f66a2d55a9604f40ff3 Cannot connect to remote host smithi038 [7285958] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285958/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285958/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d3e032a0c514ce2ab4b265970c5d88c Cannot connect to remote host smithi177 Dead ================================================================= [7285853] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285853/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285853/ Error reimaging machines: Failed to power on smithi183 [7285894] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285894/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285894/ Error reimaging machines: Failed to power on smithi151 [7285895] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:03:13 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285895/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285895/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a9eb97e254d483eb56474c4fe051f5d {'smithi027.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7285896] rados/standalone/{supported-random-distro$/{centos_8} workloads/misc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285896/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285896/ Error reimaging machines: Failed to power on smithi146 [7285897] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/dedup-io-mixed} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285897/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285897/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [7285898] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mostlyread} ----------------------------------------------------------------- time: 00:03:57 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285898/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285898/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40a7770b5e784a8d8056d742ec4c510e {'smithi124.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}, 'smithi105.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7285899] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} ----------------------------------------------------------------- time: 00:04:13 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285899/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ca42122ce8f49de92f6796675f6ff64 {'smithi100.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7285900] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285900/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285900/ Error reimaging machines: Failed to power on smithi162 [7285901] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:03:37 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285901/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285901/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40676909bda3469a9e5cd6d4574a6999 {'smithi107.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['rpm', '-e', 'ceph-debuginfo'], 'delta': '0:00:00.012461', 'end': '2023-05-25 02:11:42.006067', 'invocation': {'module_args': {'_raw_params': 'rpm -e ceph-debuginfo', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 1, 'start': '2023-05-25 02:11:41.993606', 'stderr': 'error: Failed dependencies:\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mds-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-cephfs-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-radosgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) libcephfs2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-mirror-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-base-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-test-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-osd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-nbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-common-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mgr-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librbd1-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-immutable-object-cache- debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados-devel- debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-fuse- debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mon- debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rados- debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', 'stderr_lines': ['error: Failed dependencies:', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mds- debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-cephfs- debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-radosgw- debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) libcephfs2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-mirror-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-base-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-test-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-osd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados2-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-nbd-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-common-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mgr-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librbd1-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rgw-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-immutable-object-cache-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) librados-devel-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) rbd-fuse-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) ceph-mon-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64', '\tceph- debuginfo(x86-64) = 2:17.2.6-451.g1d9bb805.el8 is needed by (installed) python3-rados-debuginfo-2:17.2.6-451.g1d9bb805.el8.x86_64'], 'stdout': '', 'stdout_lines': []}, 'smithi159.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7285902] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:03:56 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285902/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285902/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da9643895ce84595a00049da730051bc {'smithi050.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7285903] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:03:13 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285903/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285903/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db9998ff62fe401d8e376a5a4a2341a6 {'smithi151.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7285908] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285908/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285908/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53279be8e38248d8bb4aa558fa8391ca SSH connection to smithi162 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' [7285910] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285910/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285910/ Error reimaging machines: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/282/task [7285933] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285933/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285933/ Error reimaging machines: Failed to power on smithi083 [7285943] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285943/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285943/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [7285946] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285946/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285946/ Error reimaging machines: Failed to power on smithi083 [7285947] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:02:04 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285947/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285947/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5af2b26cfbb64f4f88d81822477d752b {'smithi032.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7285948] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/repair_test} ----------------------------------------------------------------- time: 00:02:21 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285948/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285948/ sentry: https://sentry.ceph.com/organizations/ceph/?query=156422be477246a59a168386b10950f5 {'smithi073.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}, 'smithi110.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7285952] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285952/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285952/ Error reimaging machines: Failed to power on smithi112 [7285961] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285961/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285961/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds Pass ================================================================= [7285642] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:15:16 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285642/ [7285643] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:23:06 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285643/ [7285644] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:14:46 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285644/ [7285645] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:53 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285645/ [7285646] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} time: 02:40:13 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285646/ [7285647] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:26:03 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285647/ [7285648] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:17:42 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285648/ [7285649] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:33 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285649/ [7285650] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:30:18 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285650/ [7285651] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/repair_test} time: 00:22:08 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285651/ [7285652] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:24:22 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285652/ [7285653] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:11:45 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285653/ [7285654] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:15:42 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285654/ [7285655] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:15:30 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285655/ [7285656] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:53 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285656/ [7285657] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:11:51 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285657/ [7285658] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:28:58 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285658/ [7285659] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_5925} time: 00:19:07 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285659/ [7285660] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:17:03 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285660/ [7285661] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/valgrind} time: 01:04:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285661/ [7285662] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:19:37 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285662/ [7285663] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{filestore-xfs} supported-random-distro$/{rhel_8} tasks/progress} time: 00:25:07 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285663/ [7285664] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:12:46 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285664/ [7285665] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:27:50 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285665/ [7285666] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:25:51 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285666/ [7285667] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:16 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285667/ [7285668] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:38:14 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285668/ [7285670] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:37:28 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285670/ [7285671] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:17:19 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285671/ [7285674] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:25:11 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285674/ [7285675] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:30:42 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285675/ [7285676] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:25:07 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285676/ [7285677] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:17:51 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285677/ [7285678] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:12:05 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285678/ [7285679] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/scrub_test} time: 00:16:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285679/ [7285680] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:15:48 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285680/ [7285681] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:44 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285681/ [7285682] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:34:21 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285682/ [7285683] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:19:44 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285683/ [7285684] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache} time: 00:14:08 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285684/ [7285685] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:17:04 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285685/ [7285686] rados/standalone/{supported-random-distro$/{centos_8} workloads/osd} time: 02:38:53 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285686/ [7285687] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:15:39 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285687/ [7285688] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:12:46 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285688/ [7285689] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli} time: 00:20:29 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285689/ [7285690] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/dedup-io-mixed} time: 00:20:25 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285690/ [7285691] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:19:31 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285691/ [7285692] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_api_tests} time: 00:30:54 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285692/ [7285693] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/prometheus} time: 00:20:09 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285693/ [7285694] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/lockdep} time: 00:21:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285694/ [7285695] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:19:59 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285695/ [7285696] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:16:44 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285696/ [7285697] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 02:36:20 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285697/ [7285698] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:11:21 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285698/ [7285699] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:19:45 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285699/ [7285700] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:34:49 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285700/ [7285701] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} time: 00:21:37 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285701/ [7285702] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:16:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285702/ [7285703] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:18:24 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285703/ [7285704] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:19:15 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285704/ [7285705] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:16:31 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285705/ [7285706] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:24:34 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285706/ [7285707] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:23:42 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285707/ [7285708] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:26:37 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285708/ [7285709] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_orch_cli_mon} time: 00:28:49 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285709/ [7285710] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:27:11 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285710/ [7285711] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:22:59 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285711/ [7285712] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:35:33 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285712/ [7285713] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:17:08 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285713/ [7285714] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:33:53 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285714/ [7285715] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:26:13 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285715/ [7285716] rados/standalone/{supported-random-distro$/{rhel_8} workloads/scrub} time: 02:28:21 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285716/ [7285717] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:15:29 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285717/ [7285718] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:10:54 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285718/ [7285719] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:17:25 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285719/ [7285720] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:20:43 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285720/ [7285721] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:24:07 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285721/ [7285722] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:10:12 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285722/ [7285723] rados/objectstore/{backends/objectstore-filestore-memstore supported-random-distro$/{centos_8}} time: 00:34:12 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285723/ [7285724] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_mon_osdmap_prune} time: 00:50:57 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285724/ [7285725] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/rados_api_tests} time: 00:35:01 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285725/ [7285726] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind} time: 00:33:58 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285726/ [7285727] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:29:33 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285727/ [7285728] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:14:59 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285728/ [7285729] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:24 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285729/ [7285730] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:09:30 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285730/ [7285731] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:25:07 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285731/ [7285732] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:05:35 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285732/ [7285733] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:51:05 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285733/ [7285734] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/off mon_election/classic task/test_adoption} time: 00:15:06 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285734/ [7285735] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:30:39 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285735/ [7285736] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:22:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285736/ [7285737] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:32:14 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285737/ [7285738] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect} time: 00:22:27 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285738/ [7285739] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:24:26 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285739/ [7285740] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:11:06 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285740/ [7285741] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:20:39 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285741/ [7285742] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:16:33 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285742/ [7285743] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:04 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285743/ [7285744] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:14:47 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285744/ [7285746] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:07 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285746/ [7285747] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285747/ [7285748] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:20:28 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285748/ [7285749] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:28:48 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285749/ [7285750] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:16:06 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285750/ [7285751] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-lz4} tasks/dashboard} time: 01:07:03 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285751/ [7285752] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{rhel_8} tasks/crash} time: 00:19:05 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285752/ [7285753] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} time: 00:12:59 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285753/ [7285754] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:16:20 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285754/ [7285756] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:22:48 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285756/ [7285757] rados/standalone/{supported-random-distro$/{rhel_8} workloads/c2c} time: 00:17:43 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285757/ [7285759] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:26:53 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285759/ [7285760] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm} time: 00:18:56 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285760/ [7285761] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:17:22 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285761/ [7285762] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:29:19 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285762/ [7285763] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep} time: 00:26:58 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285763/ [7285764] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:18:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285764/ [7285765] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:25 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285765/ [7285766] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:24:27 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285766/ [7285767] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:12 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285767/ [7285768] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:19:52 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285768/ [7285769] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:09:16 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285769/ [7285770] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:15:23 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285770/ [7285771] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/basic 3-final} time: 00:12:26 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285771/ [7285772] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:29:12 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285772/ [7285773] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:27:35 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285773/ [7285774] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:23:08 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285774/ [7285775] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:19:55 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285775/ [7285776] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:22:22 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285776/ [7285777] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_cephadm_repos} time: 00:08:21 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285777/ [7285778] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/rados_python} time: 00:19:20 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285778/ [7285779] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:58 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285779/ [7285780] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:19:49 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285780/ [7285781] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:13:33 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285781/ [7285782] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:14:05 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285782/ [7285783] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:29:03 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285783/ [7285784] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8}} time: 00:20:19 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285784/ [7285785] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:08:44 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285785/ [7285786] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{rhel_8} tasks/failover} time: 00:17:44 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285786/ [7285787] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:23:46 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285787/ [7285788] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:09:33 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285788/ [7285789] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:52 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285789/ [7285790] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:29:29 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285790/ [7285791] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:11:03 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285791/ [7285792] rados/standalone/{supported-random-distro$/{rhel_8} workloads/crush} time: 00:31:21 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285792/ [7285793] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects} time: 00:29:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285793/ [7285794] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:34:07 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285794/ [7285795] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} time: 00:56:05 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285795/ [7285796] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:21:01 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285796/ [7285797] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:09:57 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285797/ [7285798] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:15:58 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285798/ [7285799] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:55 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285799/ [7285800] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:59 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285800/ [7285801] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:12:42 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285801/ [7285802] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_stress_watch} time: 00:18:47 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285802/ [7285803] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:17:03 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285803/ [7285804] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:10:42 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285804/ [7285805] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:30:22 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285805/ [7285806] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:27:11 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285806/ [7285808] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:13:31 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285808/ [7285809] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:36 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285809/ [7285810] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:32:19 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285810/ [7285811] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:14:21 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285811/ [7285812] rados/objectstore/{backends/filejournal supported-random-distro$/{rhel_8}} time: 00:13:42 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285812/ [7285813] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:25 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285813/ [7285814] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:13:26 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285814/ [7285815] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:26:32 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285815/ [7285816] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:08:35 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285816/ [7285817] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:13:29 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285817/ [7285818] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:31:25 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285818/ [7285820] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli} time: 00:18:56 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285820/ [7285821] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:21:54 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285821/ [7285822] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/one workloads/snaps-few-objects} time: 00:39:21 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285822/ [7285823] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:55 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285823/ [7285824] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:17:47 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285824/ [7285825] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_striper} time: 00:15:19 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285825/ [7285826] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:37:24 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285826/ [7285827] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:15:08 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285827/ [7285828] rados/standalone/{supported-random-distro$/{centos_8} workloads/erasure-code} time: 01:26:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285828/ [7285829] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 01:18:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285829/ [7285830] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} time: 00:15:42 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285830/ [7285831] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:15 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285831/ [7285832] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:47:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285832/ [7285833] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:09:35 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285833/ [7285834] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:19:23 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285834/ [7285837] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:18:34 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285837/ [7285838] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-radosbench} time: 01:07:25 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285838/ [7285839] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:56:36 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285839/ [7285840] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} time: 01:38:20 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285840/ [7285841] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:29:23 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285841/ [7285842] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli_mon} time: 00:36:01 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285842/ [7285843] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:11 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285843/ [7285844] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{rhel_8}} time: 02:19:18 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285844/ [7285845] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:21:50 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285845/ [7285846] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:15:13 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285846/ [7285847] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{filestore-xfs} supported-random-distro$/{rhel_8} tasks/module_selftest} time: 00:35:04 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285847/ [7285848] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_3.0} 1-start 2-services/rgw 3-final} time: 00:18:12 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285848/ [7285849] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:15:39 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285849/ [7285850] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:18:21 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285850/ [7285851] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:33:04 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285851/ [7285852] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:24:41 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285852/ [7285854] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} time: 00:33:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285854/ [7285855] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:17:41 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285855/ [7285856] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:10:11 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285856/ [7285857] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/pool-create-delete} time: 00:21:23 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285857/ [7285858] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:26:28 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285858/ [7285860] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:09:56 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285860/ [7285861] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:17:24 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285861/ [7285862] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:40:20 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285862/ [7285863] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:22:07 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285863/ [7285864] rados/standalone/{supported-random-distro$/{rhel_8} workloads/mgr} time: 00:19:45 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285864/ [7285865] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:25:13 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285865/ [7285866] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:34:20 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285866/ [7285867] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:34:07 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285867/ [7285868] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:24:34 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285868/ [7285869] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} time: 00:26:08 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285869/ [7285870] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:15:37 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285870/ [7285871] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:25:23 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285871/ [7285872] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{rhel_8}} time: 02:23:20 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285872/ [7285873] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:26:28 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285873/ [7285874] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:23 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285874/ [7285875] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:21:53 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285875/ [7285876] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_adoption} time: 00:14:54 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285876/ [7285877] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:11:27 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285877/ [7285878] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:24:54 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285878/ [7285879] rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:22:56 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285879/ [7285880] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:23:58 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285880/ [7285881] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:16:10 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285881/ [7285882] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:14 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285882/ [7285883] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:20:44 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285883/ [7285884] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:27:22 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285884/ [7285885] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:14:50 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285885/ [7285886] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:29:28 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285886/ [7285887] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:44 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285887/ [7285888] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_5925} time: 00:16:22 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285888/ [7285889] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:16:28 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285889/ [7285890] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:01 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285890/ [7285891] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache} time: 00:21:03 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285891/ [7285892] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:21:34 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285892/ [7285893] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:10:25 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285893/ [7285904] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:36:03 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285904/ [7285905] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_8} tasks/progress} time: 00:23:51 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285905/ [7285906] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 01:08:29 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285906/ [7285911] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:12:00 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285911/ [7285912] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:19:29 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285912/ [7285913] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm_repos} time: 00:10:19 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285913/ [7285914] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:29:52 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285914/ [7285915] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} tasks/dashboard} time: 01:52:09 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285915/ [7285917] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:18:42 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285917/ [7285919] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:24:04 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285919/ [7285920] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} time: 03:17:09 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285920/ [7285921] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 01:14:01 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285921/ [7285922] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_api_tests} time: 00:33:13 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285922/ [7285923] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 1-start 2-services/basic 3-final} time: 00:14:40 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285923/ [7285924] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:09:37 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285924/ [7285925] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:43:40 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285925/ [7285926] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/readwrite} time: 00:15:22 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285926/ [7285927] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8}} time: 01:03:40 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285927/ [7285928] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:45:02 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285928/ [7285929] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:48:11 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285929/ [7285930] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 01:10:20 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285930/ [7285931] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 01:26:29 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285931/ [7285932] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:16:14 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285932/ [7285934] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:36:14 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285934/ [7285935] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:20:28 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285935/ [7285936] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} time: 01:12:03 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285936/ [7285937] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:22:19 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285937/ [7285938] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:21:54 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285938/ [7285939] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{filestore-xfs} supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:12:02 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285939/ [7285940] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:37 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285940/ [7285941] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/radosbench} time: 02:05:30 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285941/ [7285942] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:21:41 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285942/ [7285950] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:24:39 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285950/ [7285951] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:55:40 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285951/ [7285953] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_mon_osdmap_prune} time: 01:05:56 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285953/ [7285954] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:16 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285954/ [7285957] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:56:12 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285957/ [7285959] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:53:15 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285959/ [7285960] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_8}} time: 01:04:02 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285960/ [7285962] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:55 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285962/ [7285963] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-small-objects} time: 00:57:18 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285963/ [7285964] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:21:38 info: http://pulpito.ceph.com/yuriw-2023-05-24_23:03:22-rados-wip-yuri3-testing-2023-05-24-1136-quincy-distro-default-smithi/7285964/