2023-07-09T10:42:08.989 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-09T10:42:08.990 INFO:teuthology.results:subset: '111/120000' 2023-07-09T10:42:08.990 INFO:teuthology.results:seed: '3936' 2023-07-09T14:05:50.640 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-09T14:05:50.676 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-09T17:02:12.237 INFO:teuthology.results:Tests finished! gathering results... 2023-07-09T17:02:33.828 INFO:scrape:Found 307 jobs 2023-07-09T17:02:34.107 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330425/teuthology.log 2023-07-09T17:02:36.485 INFO:scrape:Found 9 distinct failure reasons 2023-07-09T17:02:36.486 INFO:scrape:Failure: No module named 'tasks.ceph' 2023-07-09T17:02:36.487 INFO:scrape:7 jobs: ['7330343', '7330358', '7330355', '7330361', '7330360', '7330342', '7330354'] 2023-07-09T17:02:36.487 INFO:scrape:suites intersection: [] 2023-07-09T17:02:36.488 INFO:scrape:suites union: ['agent/on', 'ceph', 'clusters/{fixed-2', 'clusters/{fixed-4', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/objectstore/{backends/fusestore', 'rados/singleton-nomsgr/{all/crushdiff', 'rados/singleton/{all/mon-config', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{default}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_20.04}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'task/test_rgw_multisite}', 'tasks/rados_python}', 'thrashers/default', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2023-07-09T17:02:36.489 INFO:scrape: 2023-07-09T17:02:36.489 INFO:scrape:Failure: Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi131 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2a555d3f656c6f10449c1c0fa097caa31f4056c5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' 2023-07-09T17:02:36.490 INFO:scrape:2 jobs: ['7330523', '7330476'] 2023-07-09T17:02:36.490 INFO:scrape:suites intersection: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'rados', 'test_envlibrados_for_rocksdb}'] 2023-07-09T17:02:36.491 INFO:scrape:suites union: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb/{supported/centos_latest', 'rados/singleton/{all/test_envlibrados_for_rocksdb/{supported/rhel_latest', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_20.04}}', 'test_envlibrados_for_rocksdb}'] 2023-07-09T17:02:36.491 INFO:scrape: 2023-07-09T17:02:36.491 INFO:scrape:Dead: None 2023-07-09T17:02:36.492 INFO:scrape:40 jobs 2023-07-09T17:02:36.492 INFO:scrape: 2023-07-09T17:02:36.492 INFO:scrape:Failure: {'smithi007.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'only_upgrade': False, 'package': None, 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': True, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'msg': 'Failed to update apt cache: unknown reason'}} 2023-07-09T17:02:36.493 INFO:scrape:1 jobs: ['7330433'] 2023-07-09T17:02:36.493 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'no_pools', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/multimon/{clusters/6', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/mon_clock_with_skews}'] 2023-07-09T17:02:36.493 INFO:scrape: 2023-07-09T17:02:36.494 INFO:scrape:Failure: Command failed on smithi154 with status 100: 'sudo apt-get -y --force-yes install python3-yaml python3-lxml librbd-dev collectl' 2023-07-09T17:02:36.494 INFO:scrape:3 jobs: ['7330348', '7330365', '7330351'] 2023-07-09T17:02:36.494 INFO:scrape:suites intersection: [] 2023-07-09T17:02:36.495 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/bluestore-bitmap', 'openstack', 'openstack}', 'rados', 'rados/perf/{ceph', 'rados/singleton-nomsgr/{all/export-after-evict', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'scheduler/dmclock_1Shard_16Threads', 'settings/optimized', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}}', 'thrashers/careful', 'thrashosds-health', 'ubuntu_latest', 'workloads/radosbench_4K_seq_read}', 'workloads/redirect_promote_tests}'] 2023-07-09T17:02:36.495 INFO:scrape: 2023-07-09T17:02:36.495 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2023-07-09T17:02:36.495 INFO:scrape:1 jobs: ['7330401'] 2023-07-09T17:02:36.496 INFO:scrape:suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] 2023-07-09T17:02:36.496 INFO:scrape: 2023-07-09T17:02:36.496 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2a555d3f656c6f10449c1c0fa097caa31f4056c5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2023-07-09T17:02:36.497 INFO:scrape:1 jobs: ['7330392'] 2023-07-09T17:02:36.497 INFO:scrape:suites: ['debug/mgr', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-snappy}', 'tasks/e2e}'] 2023-07-09T17:02:36.497 INFO:scrape: 2023-07-09T17:02:36.498 INFO:scrape:Dead: b"teuthology.exceptions.CommitNotFoundError: '2a555d3f656c6f10449c1c0fa097caa31f4056c5' not found in repo: https://github.com/ceph/ceph-ci.git!" 2023-07-09T17:02:36.498 INFO:scrape:1 jobs: ['7330382'] 2023-07-09T17:02:36.498 INFO:scrape: 2023-07-09T17:02:36.498 INFO:scrape:Failure: {'smithi154.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}, 'smithi045.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} 2023-07-09T17:02:36.499 INFO:scrape:2 jobs: ['7330493', '7330335'] 2023-07-09T17:02:36.499 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'mon_election/classic', 'openstack}', 'rados', 'thrashosds-health'] 2023-07-09T17:02:36.500 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'fast/fast', 'mon_election/classic', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{default}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'thrashers/morepggrow', 'thrashers/none', 'thrashosds-health', 'workloads/cache-snaps}', 'workloads/ec-small-objects-fast-read-overwrites}'] 2023-07-09T17:02:36.500 INFO:scrape: 2023-07-09T17:02:36.512 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/ failed: 14 dead: 44 running: 0 waiting: 0 queued: 0 passed: 249 Fail ================================================================= [7330342] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:15:21 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330342/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca6021f7a43f4e7cb54633a11706b8f3 No module named 'tasks.cephadm' [7330343] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:21:15 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330343/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad3b57d4f00e43f1a04e0595adcbba32 No module named 'tasks.ceph' [7330348] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} ----------------------------------------------------------------- time: 00:11:17 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330348/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330348/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c49984f7b34644f4be3208011c73b9ce Command failed on smithi154 with status 100: 'sudo apt-get -y --force-yes install python3-yaml python3-lxml librbd-dev collectl' [7330351] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:11:05 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330351/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330351/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4b3a9b1130c44cf89af14d7be3e2b5d Command failed on smithi092 with status 1: 'sudo yum -y install cephfs- java' [7330354] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} ----------------------------------------------------------------- time: 00:11:10 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330354/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330354/ sentry: https://sentry.ceph.com/organizations/ceph/?query=505d1e4f9d9f4318b23416fe9ede3d88 No module named 'tasks.ceph' [7330355] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:11:50 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330355/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330355/ sentry: https://sentry.ceph.com/organizations/ceph/?query=36906c731dab4042999aec57c0d4e0e2 No module named 'tasks.thrashosds' [7330358] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} ----------------------------------------------------------------- time: 00:08:58 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330358/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330358/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af16a14579df48f498151031fbf5fcf4 No module named 'tasks.ceph' [7330360] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:12 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330360/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330360/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a16f65fe662d4fb9afb16a0e74d9ad87 No module named 'tasks.workunit' [7330361] rados/singleton/{all/mon-config mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:51 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330361/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330361/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5a269e5f0adf4b329953c2af6d5ab06c No module named 'tasks.ceph' [7330365] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:11:48 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330365/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330365/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e375643048b741bd92d9e5ab4ef69e02 Command failed on smithi107 with status 1: 'sudo yum -y install cephfs- java' [7330392] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} tasks/e2e} ----------------------------------------------------------------- time: 00:28:49 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330392/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330392/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c2172d0f1ed4b539cf6f9825f89e97c Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2a555d3f656c6f10449c1c0fa097caa31f4056c5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7330401] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:11:46 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330401/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a7853c9ca2144a69ae809efff24318d9 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7330476] rados/singleton/{all/test_envlibrados_for_rocksdb/{supported/centos_latest test_envlibrados_for_rocksdb} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:17:41 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330476/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330476/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f995aa0a1194d039f3f096ffc5885ac Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi141 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2a555d3f656c6f10449c1c0fa097caa31f4056c5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' [7330523] rados/singleton/{all/test_envlibrados_for_rocksdb/{supported/rhel_latest test_envlibrados_for_rocksdb} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04}} ----------------------------------------------------------------- time: 00:10:24 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330523/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330523/ sentry: https://sentry.ceph.com/organizations/ceph/?query=196891dd0de74215ab2f980654353c10 Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi131 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2a555d3f656c6f10449c1c0fa097caa31f4056c5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' Dead ================================================================= [7330271] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330271/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330271/ Error reimaging machines: Failed to power on smithi003 [7330299] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330299/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330299/ Error reimaging machines: Failed to power on smithi117 [7330300] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330300/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330300/ Error reimaging machines: Expected smithi079's OS to be rhel 8.6 but found ubuntu 20.04 [7330301] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330301/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330301/ Error reimaging machines: Expected smithi093's OS to be ubuntu 22.04 but found ubuntu 20.04 [7330303] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330303/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330303/ Error reimaging machines: Expected smithi173's OS to be centos 8 but found ubuntu 22.04 [7330304] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330304/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330304/ Error reimaging machines: Expected smithi183's OS to be ubuntu 22.04 but found centos 8 [7330305] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330305/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330305/ Error reimaging machines: Expected smithi097's OS to be ubuntu 22.04 but found rhel 8.6 [7330327] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330327/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330327/ Error reimaging machines: Cannot connect to remote host smithi031 [7330328] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330328/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330328/ Error reimaging machines: Cannot connect to remote host smithi121 [7330329] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/one workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330329/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330329/ Error reimaging machines: Failed to power on smithi131 [7330330] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} tasks/rados_cls_all} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330330/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330330/ Error reimaging machines: Expected smithi005's OS to be ubuntu 20.04 but found rhel 8.6 [7330331] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330331/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330331/ Error reimaging machines: Expected smithi181's OS to be centos 8 but found ubuntu 20.04 [7330332] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330332/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330332/ Error reimaging machines: Cannot connect to remote host smithi040 [7330333] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04} tasks/mon_clock_with_skews} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330333/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330333/ Error reimaging machines: Expected smithi186's OS to be ubuntu 20.04 but found ubuntu 22.04 [7330334] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} thrashers/none thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330334/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330334/ Error reimaging machines: Expected smithi061's OS to be ubuntu 20.04 but found ubuntu 22.04 [7330335] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:04:39 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330335/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330335/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9ba3000bc164b4599d5d4d051090bd5 {'smithi179.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}, 'smithi017.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7330341] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330341/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330341/ Error reimaging machines: Failed to power on smithi098 [7330371] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330371/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330371/ Error reimaging machines: Failed to power on smithi017 [7330380] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330380/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330380/ Error reimaging machines: Failed to power on smithi057 [7330382] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330382/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330382/ [7330425] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330425/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330425/ Error reimaging machines: Cannot connect to remote host smithi172 [7330426] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330426/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330426/ Error reimaging machines: Failed to power on smithi179 [7330427] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330427/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330427/ Error reimaging machines: Cannot connect to remote host smithi164 [7330428] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330428/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330428/ Error reimaging machines: Expected smithi177's OS to be rhel 8.6 but found centos 8 [7330429] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330429/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330429/ Error reimaging machines: Expected smithi186's OS to be rhel 8.6 but found centos 8 [7330433] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} tasks/mon_clock_with_skews} ----------------------------------------------------------------- time: 00:18:24 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330433/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330433/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4aa67d6e0e9b491885339cab1660edb8 {'smithi007.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'only_upgrade': False, 'package': None, 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': True, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'msg': 'Failed to update apt cache: unknown reason'}} [7330442] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330442/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330442/ Error reimaging machines: Failed to power on smithi083 [7330452] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330452/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330452/ Error reimaging machines: Expected smithi077's OS to be rhel 8.6 but found centos 8 [7330453] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330453/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330453/ Error reimaging machines: 'ssh_keyscan smithi077.front.sepia.ceph.com' reached maximum tries (6) after waiting for 5 seconds [7330474] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330474/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330474/ Error reimaging machines: Failed to power on smithi069 [7330489] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_8} tasks/prometheus} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330489/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330489/ Error reimaging machines: Failed to power on smithi183 [7330490] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330490/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330490/ Error reimaging machines: Expected smithi184's OS to be rhel 8.6 but found ubuntu 22.04 [7330491] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330491/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330491/ Error reimaging machines: Cannot connect to remote host smithi046 [7330493] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:03:23 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330493/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330493/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4c029deee7c42c4a6e5b558ba922d71 {'smithi154.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}, 'smithi045.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7330524] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330524/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330524/ Error reimaging machines: Cannot connect to remote host smithi132 [7330525] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330525/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330525/ Error reimaging machines: Failed to power on smithi152 [7330526] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330526/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330526/ Error reimaging machines: Expected smithi037's OS to be centos 8 but found ubuntu 22.04 [7330527] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330527/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330527/ Error reimaging machines: Expected smithi184's OS to be ubuntu 20.04 but found rhel 8.6 [7330528] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330528/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330528/ Error reimaging machines: Expected smithi186's OS to be centos 8 but found rhel 8.6 [7330533] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330533/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330533/ Error reimaging machines: Failed to power on smithi139 [7330534] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330534/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330534/ Error reimaging machines: Failed to power on smithi082 [7330545] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330545/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330545/ Error reimaging machines: Failed to power on smithi107 [7330546] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zstd} tasks/e2e} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330546/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330546/ Error reimaging machines: Cannot connect to remote host smithi148 [7330565] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330565/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330565/ Error reimaging machines: Failed to power on smithi151 Pass ================================================================= [7330259] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} time: 00:41:59 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330259/ [7330260] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} time: 00:33:27 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330260/ [7330261] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:14:16 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330261/ [7330262] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:58 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330262/ [7330263] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:38:43 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330263/ [7330264] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_cephadm_repos} time: 00:09:47 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330264/ [7330265] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:24:24 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330265/ [7330266] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} time: 00:19:33 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330266/ [7330267] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04}} time: 00:11:52 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330267/ [7330268] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:17:15 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330268/ [7330269] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:33:56 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330269/ [7330270] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_8}} time: 00:11:50 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330270/ [7330272] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:22:04 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330272/ [7330273] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:50 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330273/ [7330274] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:23:08 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330274/ [7330275] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:41 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330275/ [7330276] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache} time: 00:20:06 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330276/ [7330277] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:27:38 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330277/ [7330278] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/libcephsqlite} time: 00:23:34 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330278/ [7330279] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:19:46 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330279/ [7330280] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:21:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330280/ [7330281] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} time: 02:04:37 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330281/ [7330282] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:14:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330282/ [7330283] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:36:51 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330283/ [7330284] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:18 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330284/ [7330285] rados/standalone/{supported-random-distro$/{rhel_8} workloads/scrub} time: 02:45:53 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330285/ [7330286] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{rhel_8} tasks/workunits} time: 00:15:58 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330286/ [7330287] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:11 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330287/ [7330288] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/dedup-io-mixed} time: 00:24:46 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330288/ [7330289] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:25:18 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330289/ [7330290] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:22:07 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330290/ [7330291] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:12:15 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330291/ [7330292] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:34:27 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330292/ [7330293] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_nfs} time: 00:43:45 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330293/ [7330294] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-snaps} time: 00:17:07 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330294/ [7330295] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:11:45 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330295/ [7330296] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:42:02 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330296/ [7330297] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:31:47 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330297/ [7330298] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:24:18 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330298/ [7330302] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_orch_cli} time: 00:20:21 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330302/ [7330306] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:34:35 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330306/ [7330307] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:39:34 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330307/ [7330308] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:03 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330308/ [7330309] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330309/ [7330310] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:36:22 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330310/ [7330311] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/rados_api_tests} time: 00:36:04 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330311/ [7330312] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_20.04}} time: 00:09:36 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330312/ [7330313] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-snappy} tasks/dashboard} time: 01:11:05 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330313/ [7330314] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_8} tasks/crash} time: 00:20:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330314/ [7330315] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8}} time: 00:41:45 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330315/ [7330316] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:16:42 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330316/ [7330317] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} time: 00:18:26 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330317/ [7330318] rados/standalone/{supported-random-distro$/{rhel_8} workloads/c2c} time: 00:23:14 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330318/ [7330319] rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:31:24 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330319/ [7330320] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:31:10 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330320/ [7330321] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:22:11 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330321/ [7330322] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli_mon} time: 00:30:26 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330322/ [7330323] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:21:12 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330323/ [7330324] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:14:34 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330324/ [7330325] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:34:36 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330325/ [7330326] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:18:27 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330326/ [7330336] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:42 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330336/ [7330337] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:35 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330337/ [7330338] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:14:57 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330338/ [7330339] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:21:28 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330339/ [7330340] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:41 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330340/ [7330344] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:16:06 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330344/ [7330345] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:13:45 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330345/ [7330346] rados/standalone/{supported-random-distro$/{centos_8} workloads/crush} time: 00:29:48 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330346/ [7330347] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:54 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330347/ [7330349] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:26 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330349/ [7330350] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:47 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330350/ [7330352] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:35:39 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330352/ [7330353] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} time: 00:53:16 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330353/ [7330356] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:17:51 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330356/ [7330357] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:29:12 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330357/ [7330359] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:24:41 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330359/ [7330362] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/pool-create-delete} time: 00:23:40 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330362/ [7330363] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:22:32 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330363/ [7330364] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:34:56 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330364/ [7330366] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_adoption} time: 00:17:40 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330366/ [7330367] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:30:36 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330367/ [7330368] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:15:43 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330368/ [7330369] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:25:21 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330369/ [7330370] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{rhel_8} tasks/insights} time: 00:24:41 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330370/ [7330372] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:24:11 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330372/ [7330373] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:14:01 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330373/ [7330374] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:35:13 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330374/ [7330375] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:31:29 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330375/ [7330376] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:30:40 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330376/ [7330377] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:25:50 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330377/ [7330378] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 01:36:31 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330378/ [7330379] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_20.04} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:04 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330379/ [7330381] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:33 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330381/ [7330383] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:18:34 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330383/ [7330384] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} time: 00:11:38 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330384/ [7330385] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:25:02 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330385/ [7330386] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:35:54 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330386/ [7330387] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:20:23 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330387/ [7330388] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} time: 00:19:54 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330388/ [7330389] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:15:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330389/ [7330390] rados/singleton/{all/peer mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:12:16 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330390/ [7330391] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:21:45 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330391/ [7330393] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} time: 00:10:42 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330393/ [7330394] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:22:07 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330394/ [7330395] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:22:26 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330395/ [7330396] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_5925} time: 00:20:16 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330396/ [7330397] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:12:23 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330397/ [7330398] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/rados_striper} time: 00:11:38 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330398/ [7330399] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:17:49 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330399/ [7330400] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:17:49 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330400/ [7330402] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_20.04} tasks/module_selftest} time: 00:27:39 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330402/ [7330403] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:46:31 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330403/ [7330404] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:19:36 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330404/ [7330405] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:19:54 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330405/ [7330406] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:17:42 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330406/ [7330407] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:18:53 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330407/ [7330408] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:15:10 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330408/ [7330409] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 01:55:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330409/ [7330410] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:50 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330410/ [7330411] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:13:33 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330411/ [7330412] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:37:08 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330412/ [7330413] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:32:20 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330413/ [7330414] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:14:59 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330414/ [7330415] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:14 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330415/ [7330416] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:42 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330416/ [7330417] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} time: 00:19:11 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330417/ [7330418] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:26:07 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330418/ [7330419] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} time: 00:36:28 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330419/ [7330420] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:42:11 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330420/ [7330421] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:25:33 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330421/ [7330422] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:28:07 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330422/ [7330423] rados/singleton/{all/radostool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:18:41 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330423/ [7330424] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_big} time: 00:33:51 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330424/ [7330430] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:17:30 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330430/ [7330431] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_20.04} tasks/per_module_finisher_stats} time: 00:11:04 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330431/ [7330432] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:25:16 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330432/ [7330434] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:24:27 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330434/ [7330435] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:19:02 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330435/ [7330436] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:17:33 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330436/ [7330437] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:27:35 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330437/ [7330438] rados/standalone/{supported-random-distro$/{rhel_8} workloads/misc} time: 01:14:16 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330438/ [7330439] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:21:05 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330439/ [7330440] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:43 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330440/ [7330441] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:17:31 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330441/ [7330443] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:23 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330443/ [7330444] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:24:32 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330444/ [7330445] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} tasks/rados_workunit_loadgen_mix} time: 00:22:16 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330445/ [7330446] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_8}} time: 00:27:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330446/ [7330447] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_orch_cli} time: 00:24:35 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330447/ [7330448] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:27:34 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330448/ [7330449] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:36:04 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330449/ [7330450] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:32:16 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330450/ [7330451] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:31:48 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330451/ [7330454] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:15:49 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330454/ [7330455] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:51 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330455/ [7330456] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:13:53 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330456/ [7330457] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:45:55 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330457/ [7330458] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{rhel_8} tasks/progress} time: 00:30:49 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330458/ [7330459] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:39:26 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330459/ [7330460] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:16:42 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330460/ [7330461] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} time: 00:12:39 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330461/ [7330462] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:30:50 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330462/ [7330463] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:57:24 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330463/ [7330464] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:25:05 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330464/ [7330465] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:25 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330465/ [7330466] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:27 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330466/ [7330467] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:32:38 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330467/ [7330468] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:35:14 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330468/ [7330469] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} tasks/dashboard} time: 01:13:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330469/ [7330470] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:26:27 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330470/ [7330471] rados/standalone/{supported-random-distro$/{ubuntu_20.04} workloads/mon-stretch} time: 00:21:08 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330471/ [7330472] rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:26:44 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330472/ [7330473] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_rgw_multisite} time: 00:19:14 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330473/ [7330475] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:13:44 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330475/ [7330477] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} time: 00:10:57 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330477/ [7330478] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:31:18 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330478/ [7330479] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:11 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330479/ [7330480] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} time: 00:18:26 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330480/ [7330481] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:38:51 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330481/ [7330482] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330482/ [7330483] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 01:04:13 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330483/ [7330484] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:24:55 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330484/ [7330485] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:16:03 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330485/ [7330486] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:22:13 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330486/ [7330487] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:19:03 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330487/ [7330488] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} time: 02:13:37 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330488/ [7330492] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:30:34 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330492/ [7330494] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04} tasks/readwrite} time: 00:14:16 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330494/ [7330495] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:17:59 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330495/ [7330496] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:16:21 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330496/ [7330497] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:28:09 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330497/ [7330498] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits} time: 00:40:50 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330498/ [7330499] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:26:36 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330499/ [7330500] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} time: 01:25:57 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330500/ [7330501] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache} time: 00:14:59 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330501/ [7330502] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:12:11 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330502/ [7330503] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_adoption} time: 00:21:15 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330503/ [7330504] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} time: 00:09:27 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330504/ [7330505] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:32:03 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330505/ [7330506] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:22:55 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330506/ [7330507] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} time: 00:14:40 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330507/ [7330508] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/dedup-io-mixed} time: 00:23:40 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330508/ [7330509] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:37:30 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330509/ [7330510] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:24:08 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330510/ [7330511] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:14:26 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330511/ [7330512] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:11:40 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330512/ [7330513] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:44 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330513/ [7330514] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:20:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330514/ [7330515] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:37:35 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330515/ [7330516] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:16:41 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330516/ [7330517] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_8} tasks/workunits} time: 00:19:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330517/ [7330518] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} time: 00:22:55 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330518/ [7330519] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:55 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330519/ [7330520] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:44 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330520/ [7330521] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:32:18 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330521/ [7330522] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:21:12 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330522/ [7330529] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:15:37 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330529/ [7330530] rados/standalone/{supported-random-distro$/{ubuntu_20.04} workloads/osd-backfill} time: 02:37:26 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330530/ [7330531] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:35:02 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330531/ [7330532] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/pool-create-delete} time: 00:26:14 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330532/ [7330535] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} tasks/mon_clock_with_skews} time: 00:09:30 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330535/ [7330536] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:37:00 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330536/ [7330537] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:27:17 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330537/ [7330538] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:14 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330538/ [7330539] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:14:13 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330539/ [7330540] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/scrub_test} time: 00:17:29 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330540/ [7330541] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm_repos} time: 00:07:40 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330541/ [7330542] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:18:29 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330542/ [7330543] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:27:13 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330543/ [7330544] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:31 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330544/ [7330547] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{rhel_8} tasks/crash} time: 00:18:44 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330547/ [7330548] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330548/ [7330549] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:19:01 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330549/ [7330550] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:39:50 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330550/ [7330551] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:27 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330551/ [7330552] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:20:34 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330552/ [7330553] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:18:31 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330553/ [7330554] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:46 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330554/ [7330555] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:00:55 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330555/ [7330556] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:17:45 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330556/ [7330557] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} time: 00:56:31 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330557/ [7330558] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:20:19 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330558/ [7330559] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_20.04}} time: 00:10:50 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330559/ [7330560] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:14:27 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330560/ [7330561] rados/standalone/{supported-random-distro$/{rhel_8} workloads/osd} time: 02:39:29 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330561/ [7330562] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/libcephsqlite} time: 00:18:44 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330562/ [7330563] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:19:03 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330563/ [7330564] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect} time: 00:15:32 info: http://pulpito.ceph.com/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330564/