2024-01-25T17:26:00.247 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-25T17:26:00.248 INFO:teuthology.results:subset: '111/120000' 2024-01-25T17:26:00.248 INFO:teuthology.results:seed: '6332' 2024-01-25T22:24:41.755 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-25T22:24:41.783 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-26T01:12:58.526 INFO:teuthology.results:Tests finished! gathering results... 2024-01-26T01:13:31.808 INFO:scrape:Found 428 jobs 2024-01-26T01:13:41.638 INFO:scrape:Found 11 distinct failure reasons 2024-01-26T01:13:41.639 INFO:scrape:Failure: Command failed on smithi100 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d338058-bbb3-11ee-95b2-87774f69a715 -e sha1=a72a5dace2353a54147d37bd2699a2a482507966 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2024-01-26T01:13:41.639 INFO:scrape:5 jobs: ['7531973', '7532049', '7532128', '7532188', '7531799'] 2024-01-26T01:13:41.639 INFO:scrape:suites intersection: [] 2024-01-26T01:13:41.639 INFO:scrape:suites union: ['1-bootstrap/octopus', '1-start', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9'] 2024-01-26T01:13:41.639 INFO:scrape: 2024-01-26T01:13:41.639 INFO:scrape:Failure: Failed to reconnect to smithi027 2024-01-26T01:13:41.640 INFO:scrape:10 jobs: ['7532455', '7532577', '7532045', '7532598', '7532150', '7532579', '7532036', '7532528', '7531911', '7531990'] 2024-01-26T01:13:41.640 INFO:scrape:suites intersection: [] 2024-01-26T01:13:41.640 INFO:scrape:suites union: ['0-nvme-loop', '1-install/luminous-v1only', '1-install/mimic', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-start', '2-partial-upgrade/firsthalf', '2-services/nfs-ingress-rgw-user', '3-final}', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-pacific', '8-workload/{rbd-python', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'bluestore-bitmap', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'distro/ubuntu_18.04', 'fixed-2', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'rados', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'rados/cephadm/smoke/{0-nvme-loop', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados/cephadm/with-work/{0-distro/ubuntu_18.04', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start', 'start}', 'tasks/rados_python}', 'thrashers/careful', 'thrashers/default', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'ubuntu_18.04}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] 2024-01-26T01:13:41.640 INFO:scrape: 2024-01-26T01:13:41.640 INFO:scrape:Failure: timeout expired in wait_until_healthy 2024-01-26T01:13:41.640 INFO:scrape:4 jobs: ['7532277', '7532359', '7532599', '7532508'] 2024-01-26T01:13:41.640 INFO:scrape:suites intersection: [] 2024-01-26T01:13:41.640 INFO:scrape:suites union: ['1-bootstrap/octopus', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-nfs', '2-repo_digest/defaut', '3-inline/yes', '3-upgrade-with-workload', '3-upgrade/simple', '4-final}', '4-verify}', '4-verify}}', '4-wait', '5-upgrade-ls', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9', 'roles', 'tasks/{0-from/v16.2.4'] 2024-01-26T01:13:41.640 INFO:scrape: 2024-01-26T01:13:41.641 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2024-01-26T01:13:41.641 INFO:scrape:5 jobs: ['7532603', '7532532', '7532373', '7532273', '7532592'] 2024-01-26T01:13:41.641 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2024-01-26T01:13:41.641 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress2', '2-services/rgw-ingress', '3-final}', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2024-01-26T01:13:41.641 INFO:scrape: 2024-01-26T01:13:41.641 INFO:scrape:Failure: Command failed on smithi187 with status 5: 'sudo systemctl stop ceph-f4b75ece-bbca-11ee-95b2-87774f69a715@mon.smithi187' 2024-01-26T01:13:41.641 INFO:scrape:7 jobs: ['7532443', '7532465', '7532542', '7532593', '7532323', '7532397', '7532327'] 2024-01-26T01:13:41.641 INFO:scrape:suites intersection: ['3-upgrade-with-workload'] 2024-01-26T01:13:41.641 INFO:scrape:suites union: ['1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-nfs', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-final}', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools', 'roles', 'tasks/{0-from/v16.2.4'] 2024-01-26T01:13:41.641 INFO:scrape: 2024-01-26T01:13:41.641 INFO:scrape:Failure: 'package_manager_version' 2024-01-26T01:13:41.642 INFO:scrape:2 jobs: ['7531941', '7532339'] 2024-01-26T01:13:41.642 INFO:scrape:suites intersection: ['cluster/{fixed-3', 'openstack}', 'pool/datapool', 'rados/cephadm/rbd_iscsi/{base/install', 'workloads/ceph_iscsi}'] 2024-01-26T01:13:41.642 INFO:scrape:suites union: ['cluster/{fixed-3', 'openstack}', 'pool/datapool', 'rados/cephadm/rbd_iscsi/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'workloads/ceph_iscsi}'] 2024-01-26T01:13:41.642 INFO:scrape: 2024-01-26T01:13:41.642 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=585bdf827237450fb80df4a856eba434a64948fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-01-26T01:13:41.642 INFO:scrape:2 jobs: ['7532587', '7532379'] 2024-01-26T01:13:41.642 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] 2024-01-26T01:13:41.642 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] 2024-01-26T01:13:41.642 INFO:scrape: 2024-01-26T01:13:41.642 INFO:scrape:Failure: "2024-01-25T22:28:32.740825+0000 mon.a (mon.0) 473 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2024-01-26T01:13:41.642 INFO:scrape:2 jobs: ['7532567', '7532066'] 2024-01-26T01:13:41.642 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/lockdep}'] 2024-01-26T01:13:41.643 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/lockdep}'] 2024-01-26T01:13:41.643 INFO:scrape: 2024-01-26T01:13:41.643 INFO:scrape:Failure: Command failed on smithi053 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a72a5dace2353a54147d37bd2699a2a482507966 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0353e8ee-bbbc-11ee-95b2-87774f69a715 -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\'' 2024-01-26T01:13:41.643 INFO:scrape:1 jobs: ['7532156'] 2024-01-26T01:13:41.643 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', 'rados/cephadm/osds/{0-distro/ubuntu_18.04'] 2024-01-26T01:13:41.643 INFO:scrape: 2024-01-26T01:13:41.643 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi106 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=585bdf827237450fb80df4a856eba434a64948fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2024-01-26T01:13:41.643 INFO:scrape:2 jobs: ['7532536', '7532319'] 2024-01-26T01:13:41.643 INFO:scrape:suites intersection: ['task/test_e2e}'] 2024-01-26T01:13:41.643 INFO:scrape:suites union: ['rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/dashboard/{0-distro/ignorelist_health', 'task/test_e2e}'] 2024-01-26T01:13:41.643 INFO:scrape: 2024-01-26T01:13:41.643 INFO:scrape:Failure: ['Failed to download key at http://download.ceph.com/keys/autobuild.asc: Request failed: '] 2024-01-26T01:13:41.644 INFO:scrape:1 jobs: ['7532059'] 2024-01-26T01:13:41.644 INFO:scrape:suites: ['1-install/luminous', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/default', 'thrashosds-health', 'three-plus-one}', 'workloads/rbd_cls}'] 2024-01-26T01:13:41.644 INFO:scrape: 2024-01-26T01:13:41.657 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/ failed: 40 dead: 1 running: 0 waiting: 0 queued: 0 passed: 387 Fail ================================================================= [7531799] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/connectivity} ----------------------------------------------------------------- time: 00:21:24 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531799/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531799/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8ea3086f7b444aabfc844082e750a93 Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e8eddf4-bbab-11ee-95b2-87774f69a715 -e sha1=a72a5dace2353a54147d37bd2699a2a482507966 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7531911] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-pacific 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531911/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531911/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18cbd963f3754bd2b8e4b1b2d274d2e0 Failed to reconnect to smithi078 [7531941] rados/cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{rhel_8} workloads/ceph_iscsi} ----------------------------------------------------------------- time: 00:20:37 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531941/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ee9df8bca8a4423945a48f39948da85 'package_manager_version' [7531973] rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} ----------------------------------------------------------------- time: 00:16:42 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531973/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531973/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20e3eb4cb86f43fdb2be53374f1e5c70 Command failed on smithi100 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d338058-bbb3-11ee-95b2-87774f69a715 -e sha1=a72a5dace2353a54147d37bd2699a2a482507966 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7531990] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531990/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e5e0ffdf2ee419c93b3ce322d3e83bc Failed to reconnect to smithi078 [7532036] rados/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532036/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532036/ sentry: https://sentry.ceph.com/organizations/ceph/?query=10033e24a2364bfba69fcf7771452f0b Failed to reconnect to smithi078 [7532045] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532045/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532045/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5dffe13da96b40d1948c36fb6a25f905 Failed to reconnect to smithi053 [7532049] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} ----------------------------------------------------------------- time: 00:37:06 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532049/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532049/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6c8672b115d4980816158e1120b81df Command failed on smithi088 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3d17a8e-bbb6-11ee-95b2-87774f69a715 -e sha1=a72a5dace2353a54147d37bd2699a2a482507966 -- bash -c \'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk \'"\'"\'{print $2}\'"\'"\')\'' [7532066] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} ----------------------------------------------------------------- time: 00:17:48 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532066/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532066/ "2024-01-25T19:33:44.294519+0000 mon.a (mon.0) 466 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7532128] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} ----------------------------------------------------------------- time: 00:23:21 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532128/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532128/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8bcd14ca89d4698b5c578d106ae2431 Command failed on smithi102 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 39c5045e-bbbb-11ee-95b2-87774f69a715 -e sha1=a72a5dace2353a54147d37bd2699a2a482507966 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7532150] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532150/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532150/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45dfcc0c153642f9a0d581340f60b6c9 Failed to reconnect to smithi134 [7532156] rados/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:15:44 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532156/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532156/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ea149e1aded4b91a15c1b9a66f461b7 Command failed on smithi053 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:a72a5dace2353a54147d37bd2699a2a482507966 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0353e8ee- bbbc-11ee-95b2-87774f69a715 -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\'' [7532188] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:17:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532188/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532188/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd188f24b6c840b09289b0ed6f269680 Command failed on smithi001 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9d604e2-bbbd-11ee-95b2-87774f69a715 -e sha1=a72a5dace2353a54147d37bd2699a2a482507966 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7532273] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:41:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532273/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532273/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73290ccb482941f592ca79b9fb5b4cfa reached maximum tries (301) after waiting for 300 seconds [7532277] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:25:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532277/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532277/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38cf59d6506242589f7a06d7b0913c1c timeout expired in wait_until_healthy [7532319] rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} ----------------------------------------------------------------- time: 00:21:23 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532319/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532319/ sentry: https://sentry.ceph.com/organizations/ceph/?query=088c193c11ed42fca8506f730b86c879 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=585bdf827237450fb80df4a856eba434a64948fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7532323] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:29 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532323/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532323/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c57fc4dc32734005821b5a460b936c53 Command failed on smithi123 with status 5: 'sudo systemctl stop ceph-8f7230ee-bbc4-11ee-95b2-87774f69a715@mon.smithi123' [7532327] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:17:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532327/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3406cd7aa30c4ce7914d2d8f6a718313 Command failed on smithi089 with status 5: 'sudo systemctl stop ceph-58840706-bbc4-11ee-95b2-87774f69a715@mon.smithi089' [7532339] rados/cephadm/rbd_iscsi/{base/install cluster/{fixed-3 openstack} pool/datapool supported-random-distro$/{centos_8} workloads/ceph_iscsi} ----------------------------------------------------------------- time: 00:11:37 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532339/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532339/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f55c3eae2eb04f2ea503be285b979e72 'package_manager_version' [7532359] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} ----------------------------------------------------------------- time: 00:19:55 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532359/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532359/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1cfd832c79e0417d96fd80ccfd228227 timeout expired in wait_until_healthy [7532373] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:41:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532373/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532373/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1cd43a87fd52460c94815009a677dce1 reached maximum tries (301) after waiting for 300 seconds [7532379] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:16:17 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532379/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532379/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5e7e12710b0489eb9a4784ca8e30db1 Command failed (workunit test cephadm/test_cephadm.sh) on smithi149 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=585bdf827237450fb80df4a856eba434a64948fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7532397] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:03 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532397/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c3186f08de44572ac85d866c748576c Command failed on smithi161 with status 5: 'sudo systemctl stop ceph-32e59c0e-bbc8-11ee-95b2-87774f69a715@mon.smithi161' [7532443] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:16 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532443/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532443/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b54273fe243442793ae6453ceb0c0c0 Command failed on smithi187 with status 5: 'sudo systemctl stop ceph-f4b75ece-bbca-11ee-95b2-87774f69a715@mon.smithi187' [7532455] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532455/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532455/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aa42eb750b164633a2c2f44632dc7d54 Failed to reconnect to smithi027 [7532465] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:17:42 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532465/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=807ba5db28b34ec18cbf4ea1744c3728 Command failed on smithi174 with status 5: 'sudo systemctl stop ceph-5b88c4a8-bbcb-11ee-95b2-87774f69a715@mon.smithi174' [7532508] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:30:26 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532508/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532508/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89c8c1ca796f461bbfa80c126901e137 timeout expired in wait_until_healthy [7532528] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532528/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532528/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b17cda701c2549c5ad14a52bba37bd1d Failed to reconnect to smithi027 [7532532] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:34:59 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532532/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532532/ sentry: https://sentry.ceph.com/organizations/ceph/?query=575da60811ed4772816fb8693fbbf3a9 reached maximum tries (301) after waiting for 300 seconds [7532536] rados/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} ----------------------------------------------------------------- time: 00:21:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532536/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532536/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e2b2b63550045ca96d1641301739586 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi106 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=585bdf827237450fb80df4a856eba434a64948fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7532542] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:47 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532542/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532542/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2351680339f84fc583793149c27afc7a Command failed on smithi167 with status 5: 'sudo systemctl stop ceph-0516a4e2-bbcf-11ee-95b2-87774f69a715@mon.smithi167' [7532567] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_cls_all validater/lockdep} ----------------------------------------------------------------- time: 00:17:49 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532567/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532567/ "2024-01-25T22:28:32.740825+0000 mon.a (mon.0) 473 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7532577] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532577/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532577/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63a3500c83da4d89a2ba3dc5b2904aae Failed to reconnect to smithi131 [7532579] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532579/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532579/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15f320e5b7d146faa29dffa8bfe92c18 Failed to reconnect to smithi146 [7532587] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:16:35 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532587/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532587/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01759e4551fb4709911c5cc4eba07df7 Command failed (workunit test cephadm/test_cephadm.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=585bdf827237450fb80df4a856eba434a64948fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7532592] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:43:25 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532592/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532592/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8de741d11dc4c58a7a5b9a4f56594dc reached maximum tries (301) after waiting for 300 seconds [7532593] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532593/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532593/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a286f90aaaa4e9594f5c37febc7b6a7 Command failed on smithi112 with status 5: 'sudo systemctl stop ceph-17bdbcae-bbd2-11ee-95b2-87774f69a715@mon.smithi112' [7532598] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532598/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532598/ sentry: https://sentry.ceph.com/organizations/ceph/?query=60cc1edb0c4348ecbac0224027bcd515 Failed to reconnect to smithi152 [7532599] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:27:43 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532599/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532599/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e49123eeee2442e9ca672097aa3104f timeout expired in wait_until_healthy [7532603] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:33:54 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532603/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532603/ sentry: https://sentry.ceph.com/organizations/ceph/?query=faddc60acf35460fa1dc6f347b53768e reached maximum tries (301) after waiting for 300 seconds Dead ================================================================= [7532059] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:05:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532059/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532059/ sentry: https://sentry.ceph.com/organizations/ceph/?query=acf88088b3444fb2b664d68c06cbbcd8 ['Failed to download key at http://download.ceph.com/keys/autobuild.asc: Request failed: '] Pass ================================================================= [7531775] rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:37:54 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531775/ [7531777] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:24:25 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531777/ [7531779] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:30:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531779/ [7531781] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.6_container_tools_rhel8 fixed-2 mon_election/connectivity start} time: 00:17:44 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531781/ [7531783] rados/objectstore/{backends/objectstore supported-random-distro$/{rhel_8}} time: 05:36:03 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531783/ [7531785] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:25:43 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531785/ [7531787] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:14:35 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531787/ [7531789] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:29 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531789/ [7531791] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:30:45 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531791/ [7531793] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:19:24 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531793/ [7531795] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/radosbench} time: 00:41:40 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531795/ [7531797] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:10 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531797/ [7531801] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:17:38 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531801/ [7531803] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_mon_osdmap_prune} time: 00:41:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531803/ [7531805] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:26:49 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531805/ [7531807] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind} time: 00:37:15 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531807/ [7531809] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:23:17 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531809/ [7531811] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:23 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531811/ [7531813] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:09:06 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531813/ [7531815] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:34:54 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531815/ [7531817] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/progress} time: 00:27:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531817/ [7531819] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:09 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531819/ [7531821] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} time: 00:28:32 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531821/ [7531823] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:16 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531823/ [7531825] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect} time: 00:30:10 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531825/ [7531827] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:21:32 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531827/ [7531829] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:16:25 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531829/ [7531831] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:38:02 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531831/ [7531833] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:22:45 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531833/ [7531834] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:39:44 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531834/ [7531836] rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:33:40 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531836/ [7531839] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:27:25 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531839/ [7531842] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} time: 00:18:42 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531842/ [7531846] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:23:23 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531846/ [7531848] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:18:30 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531848/ [7531851] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:11:55 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531851/ [7531855] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531855/ [7531857] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:18:12 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531857/ [7531860] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:25:59 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531860/ [7531863] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:06 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531863/ [7531867] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:52:55 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531867/ [7531870] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:37:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531870/ [7531872] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:22:56 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531872/ [7531875] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:54 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531875/ [7531878] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:12:17 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531878/ [7531881] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:17:58 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531881/ [7531884] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli_mon} time: 00:27:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531884/ [7531887] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:29:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531887/ [7531890] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:14:44 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531890/ [7531893] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:19:49 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531893/ [7531897] rados/dashboard/{centos_8.stream_container_tools clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/dashboard} time: 01:00:28 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531897/ [7531900] rados/objectstore/{backends/alloc-hint supported-random-distro$/{ubuntu_latest}} time: 00:09:37 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531900/ [7531902] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:18:10 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531902/ [7531905] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:19:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531905/ [7531908] rados/standalone/{supported-random-distro$/{centos_8} workloads/crush} time: 00:18:35 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531908/ [7531914] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:27:43 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531914/ [7531917] rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} time: 00:25:16 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531917/ [7531920] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:47 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531920/ [7531923] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:36:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531923/ [7531926] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep} time: 00:24:28 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531926/ [7531929] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:29:06 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531929/ [7531931] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:15:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531931/ [7531933] rados/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 2-node-mgr orchestrator_cli} time: 00:22:40 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531933/ [7531935] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:10:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531935/ [7531937] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:17 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531937/ [7531939] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:22:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531939/ [7531943] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:47 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531943/ [7531947] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:03 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531947/ [7531950] rados/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/classic start} time: 00:14:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531950/ [7531952] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:12:07 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531952/ [7531955] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:14:14 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531955/ [7531958] rados/cephadm/smoke-singlehost/{0-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} time: 00:15:21 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531958/ [7531961] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:31:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531961/ [7531964] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:29:09 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531964/ [7531967] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:32:59 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531967/ [7531970] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:25:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531970/ [7531976] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:02 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531976/ [7531979] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:20:06 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531979/ [7531982] rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:43:49 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531982/ [7531984] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:15:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531984/ [7531986] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_adoption} time: 00:11:38 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531986/ [7531988] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531988/ [7531993] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:22:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531993/ [7531996] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:21:56 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531996/ [7531999] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:21:26 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531999/ [7532002] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} time: 00:15:51 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532002/ [7532005] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:20:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532005/ [7532008] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:18:11 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532008/ [7532011] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:17:29 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532011/ [7532013] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.6_container_tools_3.0 fixed-2 mon_election/connectivity start} time: 00:21:17 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532013/ [7532015] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 01:20:16 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532015/ [7532017] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects} time: 00:20:11 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532017/ [7532019] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:34:57 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532019/ [7532021] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:11:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532021/ [7532023] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:47 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532023/ [7532024] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_mon_workunits} time: 00:38:55 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532024/ [7532025] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:12:28 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532025/ [7532026] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} time: 00:53:07 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532026/ [7532027] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:16:16 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532027/ [7532028] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm_repos} time: 00:07:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532028/ [7532029] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/workunits} time: 00:20:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532029/ [7532030] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:23:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532030/ [7532031] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:39 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532031/ [7532032] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 00:50:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532032/ [7532033] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:10:54 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532033/ [7532034] rados/standalone/{supported-random-distro$/{rhel_8} workloads/erasure-code} time: 00:49:45 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532034/ [7532035] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:40 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532035/ [7532037] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:36 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532037/ [7532038] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.6_container_tools_rhel8 fixed-2 mon_election/classic start} time: 00:18:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532038/ [7532039] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:16:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532039/ [7532040] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_stress_watch} time: 00:20:13 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532040/ [7532041] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:17:18 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532041/ [7532042] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:08:45 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532042/ [7532043] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:12:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532043/ [7532044] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:25:09 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532044/ [7532046] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:18 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532046/ [7532047] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:16:55 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532047/ [7532048] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:11 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532048/ [7532050] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:12:44 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532050/ [7532051] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:30 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532051/ [7532052] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:24:04 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532052/ [7532053] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:17:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532053/ [7532054] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:15 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532054/ [7532055] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} time: 00:25:13 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532055/ [7532056] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:25:58 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532056/ [7532057] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:21:50 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532057/ [7532058] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:18:12 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532058/ [7532060] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:20:24 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532060/ [7532061] rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:24:51 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532061/ [7532062] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:26:19 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532062/ [7532063] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:13:37 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532063/ [7532064] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:11 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532064/ [7532065] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:29:21 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532065/ [7532067] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/one workloads/snaps-few-objects} time: 00:36:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532067/ [7532068] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} time: 00:18:59 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532068/ [7532069] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/crash} time: 00:12:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532069/ [7532070] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:50 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532070/ [7532071] rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:41:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532071/ [7532072] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/rados_striper} time: 00:13:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532072/ [7532073] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:32:10 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532073/ [7532074] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} time: 00:13:40 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532074/ [7532075] rados/objectstore/{backends/filejournal supported-random-distro$/{rhel_8}} time: 00:18:02 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532075/ [7532076] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 01:43:51 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532076/ [7532077] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532077/ [7532078] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:44 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532078/ [7532079] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:15:25 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532079/ [7532082] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:11:02 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532082/ [7532083] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:23:10 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532083/ [7532086] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:24:39 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532086/ [7532088] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:12:26 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532088/ [7532090] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:19:02 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532090/ [7532092] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli_mon} time: 00:23:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532092/ [7532093] rados/standalone/{supported-random-distro$/{rhel_8} workloads/mgr} time: 00:20:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532093/ [7532095] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-radosbench} time: 01:23:56 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532095/ [7532097] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:29:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532097/ [7532100] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:50:11 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532100/ [7532102] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:16:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532102/ [7532104] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532104/ [7532106] rados/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} time: 00:21:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532106/ [7532108] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-big} time: 00:59:48 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532108/ [7532110] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532110/ [7532112] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:52 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532112/ [7532114] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:20:45 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532114/ [7532116] rados/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/connectivity start} time: 00:13:48 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532116/ [7532118] rados/cephadm/smoke-singlehost/{0-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/rgw 3-final} time: 00:17:30 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532118/ [7532120] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:34:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532120/ [7532122] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:39 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532122/ [7532124] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:15:09 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532124/ [7532126] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:33:59 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532126/ [7532130] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:23:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532130/ [7532132] rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:22:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532132/ [7532134] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:22:51 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532134/ [7532136] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_adoption} time: 00:11:04 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532136/ [7532138] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} time: 00:34:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532138/ [7532140] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:10:18 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532140/ [7532142] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:17:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532142/ [7532144] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:12:19 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532144/ [7532146] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:21:44 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532146/ [7532148] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:16:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532148/ [7532152] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:15:13 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532152/ [7532154] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:28:13 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532154/ [7532158] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:10:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532158/ [7532160] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} time: 00:16:13 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532160/ [7532162] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532162/ [7532164] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8}} time: 02:24:03 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532164/ [7532166] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:25:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532166/ [7532168] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.6_container_tools_3.0 fixed-2 mon_election/classic start} time: 00:21:28 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532168/ [7532170] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:32:43 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532170/ [7532172] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:18:49 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532172/ [7532174] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:36 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532174/ [7532176] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} time: 00:20:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532176/ [7532178] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} time: 00:32:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532178/ [7532180] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm_repos} time: 00:07:39 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532180/ [7532182] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:21:36 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532182/ [7532184] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:16:58 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532184/ [7532186] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:34:19 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532186/ [7532190] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:13:38 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532190/ [7532192] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:40:59 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532192/ [7532194] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:21:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532194/ [7532196] rados/standalone/{supported-random-distro$/{centos_8} workloads/misc} time: 00:32:48 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532196/ [7532198] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:34:21 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532198/ [7532200] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:21:07 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532200/ [7532202] rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:20:52 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532202/ [7532204] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:33:09 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532204/ [7532206] rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:47:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532206/ [7532208] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:27:36 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532208/ [7532210] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.6_container_tools_rhel8 fixed-2 mon_election/connectivity start} time: 00:17:52 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532210/ [7532212] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:10:11 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532212/ [7532214] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:13:57 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532214/ [7532216] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:50 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532216/ [7532218] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:20:23 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532218/ [7532220] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:19:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532220/ [7532222] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:24:48 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532222/ [7532224] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-snaps} time: 00:30:48 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532224/ [7532226] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:11:48 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532226/ [7532228] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:21:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532228/ [7532230] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:44 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532230/ [7532232] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_5925} time: 00:21:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532232/ [7532234] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:20:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532234/ [7532236] rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:42:29 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532236/ [7532238] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:38:37 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532238/ [7532240] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} time: 00:25:03 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532240/ [7532242] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:20:19 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532242/ [7532244] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:09:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532244/ [7532246] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:21:49 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532246/ [7532248] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache} time: 00:14:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532248/ [7532250] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:19:29 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532250/ [7532252] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:29:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532252/ [7532254] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} time: 00:48:24 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532254/ [7532256] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{ubuntu_latest}} time: 02:44:30 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532256/ [7532258] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:26:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532258/ [7532261] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:12:47 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532261/ [7532262] rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:30:51 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532262/ [7532265] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read} time: 00:21:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532265/ [7532267] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:12:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532267/ [7532269] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} time: 00:13:51 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532269/ [7532271] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:33:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532271/ [7532275] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:34:21 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532275/ [7532279] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:57 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532279/ [7532281] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:22:11 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532281/ [7532283] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:12:09 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532283/ [7532285] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:29:13 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532285/ [7532287] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/rbd_cls} time: 00:34:37 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532287/ [7532289] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:11:42 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532289/ [7532291] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:22:13 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532291/ [7532293] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:08:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532293/ [7532295] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli_mon} time: 00:23:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532295/ [7532297] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:42:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532297/ [7532299] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:18:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532299/ [7532301] rados/dashboard/{centos_8.stream_container_tools clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/dashboard} time: 00:55:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532301/ [7532303] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:19:36 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532303/ [7532305] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} time: 00:16:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532305/ [7532307] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-pacific 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_18.04} time: 02:51:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532307/ [7532309] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:50:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532309/ [7532311] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 01:03:52 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532311/ [7532313] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} time: 01:54:30 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532313/ [7532315] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:22:37 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532315/ [7532317] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:21:10 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532317/ [7532321] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:26:47 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532321/ [7532325] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:10:23 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532325/ [7532329] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:35:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532329/ [7532331] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:12:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532331/ [7532333] rados/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 2-node-mgr orchestrator_cli} time: 00:22:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532333/ [7532335] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:36:25 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532335/ [7532337] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:18:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532337/ [7532341] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:48:57 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532341/ [7532343] rados/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/classic start} time: 00:13:58 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532343/ [7532345] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:12:06 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532345/ [7532347] rados/cephadm/smoke-singlehost/{0-distro$/{ubuntu_18.04} 1-start 2-services/basic 3-final} time: 00:15:15 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532347/ [7532349] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/radosbench} time: 00:48:23 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532349/ [7532351] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:30:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532351/ [7532353] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} time: 00:30:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532353/ [7532355] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_8}} time: 00:09:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532355/ [7532357] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:09:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532357/ [7532361] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532361/ [7532363] rados/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:40:08 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532363/ [7532365] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:14 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532365/ [7532367] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_adoption} time: 00:11:03 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532367/ [7532369] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect} time: 00:15:37 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532369/ [7532371] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:19:49 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532371/ [7532375] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:43:56 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532375/ [7532377] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:15:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532377/ [7532381] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:19:03 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532381/ [7532383] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:14:26 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532383/ [7532385] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.6_container_tools_3.0 fixed-2 mon_election/connectivity start} time: 00:20:28 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532385/ [7532387] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/test_rbd_api} time: 00:33:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532387/ [7532389] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/repair_test} time: 00:17:29 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532389/ [7532391] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:18:12 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532391/ [7532393] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:09:37 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532393/ [7532395] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:12 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532395/ [7532399] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:12:59 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532399/ [7532401] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 01:07:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532401/ [7532403] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:55:00 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532403/ [7532405] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:18:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532405/ [7532407] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm_repos} time: 00:07:25 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532407/ [7532409] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:20:30 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532409/ [7532411] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532411/ [7532413] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} time: 00:35:14 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532413/ [7532415] rados/standalone/{supported-random-distro$/{centos_8} workloads/mon} time: 00:44:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532415/ [7532417] rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:30:11 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532417/ [7532419] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:21:35 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532419/ [7532421] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:23:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532421/ [7532423] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} time: 00:18:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532423/ [7532425] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:19:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532425/ [7532427] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.6_container_tools_rhel8 fixed-2 mon_election/classic start} time: 00:18:36 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532427/ [7532429] rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:16:41 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532429/ [7532431] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:08:27 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532431/ [7532433] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects} time: 00:30:05 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532433/ [7532435] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:44:35 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532435/ [7532437] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:12:24 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532437/ [7532439] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:13:42 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532439/ [7532441] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:12:13 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532441/ [7532442] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:22:35 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532442/ [7532444] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:15:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532444/ [7532445] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:24:09 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532445/ [7532446] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} time: 00:50:17 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532446/ [7532447] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:32:39 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532447/ [7532448] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:18:13 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532448/ [7532449] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/scrub_test} time: 00:12:15 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532449/ [7532451] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:22 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532451/ [7532453] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} time: 00:25:03 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532453/ [7532457] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:14:12 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532457/ [7532459] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} time: 00:28:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532459/ [7532461] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:10:07 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532461/ [7532463] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/radosbench} time: 01:01:58 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532463/ [7532467] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/progress} time: 00:19:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532467/ [7532470] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:49 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532470/ [7532473] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:32:25 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532473/ [7532476] rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:12:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532476/ [7532479] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} time: 00:57:03 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532479/ [7532482] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:20:36 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532482/ [7532485] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} time: 00:32:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532485/ [7532488] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_mon_workunits} time: 00:31:28 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532488/ [7532491] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:19:06 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532491/ [7532494] rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:34:48 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532494/ [7532497] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:20:59 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532497/ [7532500] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} time: 00:14:09 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532500/ [7532502] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:16:52 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532502/ [7532504] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:11:30 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532504/ [7532506] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:37:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532506/ [7532510] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:21 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532510/ [7532512] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:21:06 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532512/ [7532514] rados/cephadm/smoke/{0-nvme-loop distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:22:29 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532514/ [7532516] rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:19:02 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532516/ [7532518] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects} time: 00:29:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532518/ [7532520] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli_mon} time: 00:23:24 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532520/ [7532522] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:25:35 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532522/ [7532524] rados/standalone/{supported-random-distro$/{centos_8} workloads/osd} time: 03:14:10 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532524/ [7532526] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/libcephsqlite} time: 00:17:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532526/ [7532530] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:24:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532530/ [7532534] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:14:30 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532534/ [7532538] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:11:26 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532538/ [7532540] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:20 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532540/ [7532544] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:24 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532544/ [7532546] rados/cephadm/smoke/{0-nvme-loop distro/centos_8.stream_container_tools fixed-2 mon_election/connectivity start} time: 00:13:55 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532546/ [7532548] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:29:21 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532548/ [7532550] rados/cephadm/smoke-singlehost/{0-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:17:01 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532550/ [7532551] rados/singleton/{all/mon-config mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:19:47 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532551/ [7532552] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:08:49 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532552/ [7532553] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 00:42:26 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532553/ [7532554] rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} time: 00:55:58 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532554/ [7532555] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:11:19 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532555/ [7532557] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532557/ [7532559] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/prometheus} time: 00:19:35 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532559/ [7532561] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:12 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532561/ [7532563] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:25:59 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532563/ [7532565] rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:34:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532565/ [7532569] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_adoption} time: 00:11:32 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532569/ [7532571] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/snaps-few-objects} time: 00:35:50 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532571/ [7532573] rados/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:16:53 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532573/ [7532575] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:41:39 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532575/ [7532581] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:24:45 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532581/ [7532583] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:24:46 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532583/ [7532585] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:12 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532585/ [7532589] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:26:31 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532589/ [7532590] rados/cephadm/smoke/{0-nvme-loop distro/rhel_8.6_container_tools_3.0 fixed-2 mon_election/classic start} time: 00:21:23 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532590/ [7532591] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:42:21 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532591/ [7532594] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:28:25 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532594/ [7532595] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm_repos} time: 00:07:38 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532595/ [7532596] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:27:17 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532596/ [7532597] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:24:34 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532597/ [7532600] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:34:19 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532600/ [7532601] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:11:29 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532601/ [7532602] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:38 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532602/ [7532604] rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:29:33 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532604/ [7532605] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} time: 01:45:11 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532605/ [7532606] rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:41:54 info: https://pulpito.ceph.com/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532606/