2021-04-17T15:51:14.627 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-17T15:51:14.629 INFO:teuthology.results:subset: '10033/99999'
2021-04-17T15:51:14.629 INFO:teuthology.results:seed: '2477'
2021-04-17T20:06:48.687 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-17T20:06:48.734 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-04-18T08:07:21.260 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2021-04-18T08:08:04.603 INFO:scrape:Found 351 jobs
2021-04-18T08:08:57.913 INFO:scrape:Found 10 distinct failure reasons
2021-04-18T08:08:57.917 INFO:scrape:Failure: "2021-04-17T18:10:28.564747+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: rook" in cluster log
2021-04-18T08:08:57.917 INFO:scrape:108 jobs
2021-04-18T08:08:57.919 INFO:scrape:suites intersection: []
2021-04-18T08:08:57.919 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-inject-leak/mon', '2-inject-leak/none', '2-inject-leak/osd', '2-node-mgr', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'd-thrash/none', 'debug/mgr', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'orchestrator_cli}', 'rados', 'rados/basic/{ceph', 'rados/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_18.04}', 'rados/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04}', 'rados/dashboard/{clusters/{2-node-mgr}', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/multimon/{clusters/6', 'rados/singleton-bluestore/{all/cephtool', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'random-objectstore$/{bluestore-comp-zstd}', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/crash}', 'tasks/dashboard}', 'tasks/failover}', 'tasks/insights}', 'tasks/libcephsqlite}', 'tasks/module_selftest}', 'tasks/mon_clock_no_skews}', 'tasks/mon_clock_with_skews}', 'tasks/mon_recovery', 'tasks/progress}', 'tasks/prometheus}', 'tasks/rados_api_tests', 'tasks/rados_api_tests}', 'tasks/rados_cls_all', 'tasks/rados_cls_all}', 'tasks/rados_python}', 'tasks/rados_stress_watch}', 'tasks/rados_striper}', 'tasks/rados_workunit_loadgen_big}', 'tasks/rados_workunit_loadgen_mix}', 'tasks/rados_workunit_loadgen_mostlyread}', 'tasks/readwrite}', 'tasks/repair_test}', 'tasks/scrub_test}', 'tasks/workunits}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/dedup-io-mixed}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects-many-deletes}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-small-objects}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_api_tests}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/redirect_promote_tests}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}']
2021-04-18T08:08:57.920 INFO:scrape:
2021-04-18T08:08:57.921 INFO:scrape:Failure: Command failed on smithi106 with status 5: 'sudo systemctl stop ceph-3a104e70-9fb0-11eb-81bf-001a4aab830c@mon.smithi106'
2021-04-18T08:08:57.921 INFO:scrape:64 jobs
2021-04-18T08:08:57.922 INFO:scrape:suites intersection: []
2021-04-18T08:08:57.923 INFO:scrape:suites union: ['1-install/luminous', '1-install/luminous-v1only', '1-install/mimic', '1-install/mimic-v1only', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-install/octopus', '1-install/pacific', '1-start', '2-services/basic', '2-services/iscsi', '2-services/mirror', '2-services/rgw', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'rados', 'rados/cephadm/dashboard/{0-distro/centos_8.2_kubic_stable', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.2_kubic_stable', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'rados/cephadm/smoke/{distro/centos_8.2_kubic_stable', 'rados/cephadm/smoke/{distro/rhel_8.3_kubic_stable', 'rados/cephadm/smoke/{distro/ubuntu_18.04', 'rados/cephadm/smoke/{distro/ubuntu_20.04', 'rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable', 'rados/cephadm/with-work/{0-distro/centos_8.2_kubic_stable', 'rados/cephadm/with-work/{0-distro/rhel_8.3_kubic_stable', 'rados/cephadm/with-work/{0-distro/ubuntu_18.04', 'rados/cephadm/with-work/{0-distro/ubuntu_20.04', 'rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable', 'rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'root}', 'start', 'start}', 'task/test_e2e}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}']
2021-04-18T08:08:57.923 INFO:scrape:
2021-04-18T08:08:57.924 INFO:scrape:Failure: Command failed (workunit test rados/version_number_sanity.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a2e1e42f98caaf16ed5f3393512aaaefde169c3f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/version_number_sanity.sh'
2021-04-18T08:08:57.924 INFO:scrape:2 jobs: ['6053665', '6053491']
2021-04-18T08:08:57.925 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/version-number-sanity']
2021-04-18T08:08:57.925 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/version-number-sanity', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}']
2021-04-18T08:08:57.926 INFO:scrape:
2021-04-18T08:08:57.926 INFO:scrape:Dead: b'2021-04-18T05:58:28.871 DEBUG:teuthology.exit:Finished running handlers'
2021-04-18T08:08:57.927 INFO:scrape:10 jobs: ['6053475', '6053649', '6053565', '6053527', '6053507', '6053394', '6053352', '6053435', '6053613', '6053682']
2021-04-18T08:08:57.927 INFO:scrape:suites intersection: []
2021-04-18T08:08:57.928 INFO:scrape:suites union: ['1-tasks', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro1$/{rhel_8.3_kubic_stable}', 'distro1$/{ubuntu_20.04}', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.3-octopus', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']
2021-04-18T08:08:57.928 INFO:scrape:
2021-04-18T08:08:57.929 INFO:scrape:Failure: timed out waiting for admin_socket to appear after osd.0 restart
2021-04-18T08:08:57.929 INFO:scrape:1 jobs: ['6053612']
2021-04-18T08:08:57.929 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton/{all/rebuild-mondb', 'supported-random-distro$/{centos_8}}']
2021-04-18T08:08:57.930 INFO:scrape:
2021-04-18T08:08:57.930 INFO:scrape:Failure: Command failed (workunit test mon/test_mon_config_key.py) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a2e1e42f98caaf16ed5f3393512aaaefde169c3f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/test_mon_config_key.py'
2021-04-18T08:08:57.931 INFO:scrape:1 jobs: ['6053509']
2021-04-18T08:08:57.932 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/mon-config-keys', 'supported-random-distro$/{rhel_8}}']
2021-04-18T08:08:57.932 INFO:scrape:
2021-04-18T08:08:57.933 INFO:scrape:Crash: Found coredumps on ubuntu@smithi121.front.sepia.ceph.com
2021-04-18T08:08:57.933 INFO:scrape:ceph version 17.0.0-3305-ga2e1e42f (a2e1e42f98caaf16ed5f3393512aaaefde169c3f) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f040c290980]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x555e8dffb730]
 5: ceph-osd(+0xc688b0) [0x555e8dffb8b0]
 6: (PriorityCache::Manager::balance()+0x444) [0x555e8ec4b564]
 7: (BlueStore::MempoolThread::entry()+0x831) [0x555e8e68c8f1]
 8: (Thread::_entry_func(void*)+0x9) [0x555e8e7ea4a9]
 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f040c2856db]
 10: clone()
2021-04-18T08:08:57.933 INFO:scrape:1 jobs: ['6053403']
2021-04-18T08:08:57.934 INFO:scrape:suites: ['mon_election/connectivity', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/wpq_default_shards', 'settings/optimized', 'ubuntu_18.04', 'workloads/radosbench_omap_write}']
2021-04-18T08:08:57.934 INFO:scrape:
2021-04-18T08:08:57.935 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh
2021-04-18T08:08:57.935 INFO:scrape:1 jobs: ['6053575']
2021-04-18T08:08:57.936 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}']
2021-04-18T08:08:57.936 INFO:scrape:
2021-04-18T08:08:57.937 INFO:scrape:Failure: An attempt to upgrade from a higher version to a lower one will always fail. Hint: check tags in the target git branch.
2021-04-18T08:08:57.937 INFO:scrape:2 jobs: ['6053414', '6053592']
2021-04-18T08:08:57.937 INFO:scrape:suites intersection: ['frontend', 'overrides', 'rados/upgrade/pacific-x/rgw-multisite/{clusters', 'realm', 'tasks']
2021-04-18T08:08:57.938 INFO:scrape:suites union: ['frontend', 'overrides', 'rados/upgrade/pacific-x/rgw-multisite/{clusters', 'realm', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks', 'upgrade/primary}', 'upgrade/secondary}']
2021-04-18T08:08:57.938 INFO:scrape:
2021-04-18T08:08:57.939 INFO:scrape:Failure: reached maximum tries (800) after waiting for 4800 seconds
2021-04-18T08:08:57.939 INFO:scrape:1 jobs: ['6053462']
2021-04-18T08:08:57.940 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/classic', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-radosbench}']
2021-04-18T08:08:57.940 INFO:scrape:
2021-04-18T08:08:57.959 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/
failed:  181
dead:    9
running: 1
waiting: 0
queued:  0
passed:  160


Fail
=================================================================
[6053345]  rados/cephadm/smoke-roleless/{0-distro/centos_8.2_kubic_stable 1-start 2-services/basic 3-final}
-----------------------------------------------------------------
time:   00:06:34
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053345/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053345/
sentry: https://sentry.ceph.com/organizations/ceph/?query=57292f0a87a44e44b1f46afa8ab15c18

    Command failed on smithi073 with status 5: 'sudo systemctl stop
    ceph-582f8240-9f97-11eb-81bf-001a4aab830c@mon.smithi073'

[6053346]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:32:44
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053346/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053346/

    "2021-04-17T16:21:59.979556+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053347]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite}
-----------------------------------------------------------------
time:   00:15:48
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053347/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053347/

    "2021-04-17T16:19:28.256362+0000 mgr.y (mgr.4118) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053351]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench}
-----------------------------------------------------------------
time:   01:48:27
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053351/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053351/

    "2021-04-17T16:24:44.277343+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053353]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/prometheus}
-----------------------------------------------------------------
time:   00:20:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053353/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053353/

    "2021-04-17T16:30:31.778649+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053357]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:29:32
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053357/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053357/

    "2021-04-17T16:30:48.215586+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053362]  rados/cephadm/with-work/{0-distro/centos_8.2_kubic_stable fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python}
-----------------------------------------------------------------
time:   00:09:01
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053362/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053362/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cd6234cf55ea43eeacdbbcb1946c2471

    Command failed on smithi016 with status 5: 'sudo systemctl stop ceph-
    cf5e80fa-9f9c-11eb-81bf-001a4aab830c@mon.a'

[6053363]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect}
-----------------------------------------------------------------
time:   00:13:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053363/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053363/

    "2021-04-17T16:54:38.200802+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053364]  rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/snaps-few-objects}
-----------------------------------------------------------------
time:   00:05:14
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053364/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053364/
sentry: https://sentry.ceph.com/organizations/ceph/?query=61b8b74ae00f4c458d9cebc8e62f169f

    Command failed on smithi002 with status 5: 'sudo systemctl stop
    ceph-1e5710a0-9f9d-11eb-81bf-001a4aab830c@mon.a'

[6053366]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind}
-----------------------------------------------------------------
time:   00:30:43
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053366/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053366/

    "2021-04-17T17:08:44.048733+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053367]  rados/cephadm/smoke/{distro/rhel_8.3_kubic_stable fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:10:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053367/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053367/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2ee093d41fb4420b913392a6f0720f45

    Command failed on smithi012 with status 5: 'sudo systemctl stop ceph-
    cdaf263c-9f9d-11eb-81bf-001a4aab830c@mon.a'

[6053371]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests}
-----------------------------------------------------------------
time:   00:12:24
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053371/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053371/

    "2021-04-17T16:58:14.611532+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053373]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:09:24
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053373/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053373/
sentry: https://sentry.ceph.com/organizations/ceph/?query=29684f1a30a8425390550bd1bb6d32d0

    Command failed on smithi051 with status 5: 'sudo systemctl stop
    ceph-21c53450-9f9e-11eb-81bf-001a4aab830c@mon.a'

[6053374]  rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites}
-----------------------------------------------------------------
time:   00:23:51
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053374/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053374/

    "2021-04-17T16:59:00.571745+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053375]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:25:36
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053375/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053375/

    "2021-04-17T16:58:16.506023+0000 mgr.x (mgr.4112) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053378]  rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/iscsi 3-final}
-----------------------------------------------------------------
time:   00:10:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053378/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053378/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0a76326ad368407886cd17198ee23af8

    Command failed on smithi041 with status 5: 'sudo systemctl stop
    ceph-f7bf1676-9f9d-11eb-81bf-001a4aab830c@mon.smithi041'

[6053381]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object}
-----------------------------------------------------------------
time:   00:13:28
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053381/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053381/

    "2021-04-17T17:03:49.206907+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053382]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:38:46
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053382/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053382/

    "2021-04-17T17:09:12.592339+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053383]  rados/cephadm/with-work/{0-distro/rhel_8.3_kubic_stable fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:19:10
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053383/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053383/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d411055f7c5f4609b75c45c205f0b919

    Command failed on smithi038 with status 5: 'sudo systemctl stop ceph-
    ff383020-9f9f-11eb-81bf-001a4aab830c@mon.a'

[6053384]  rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest}
-----------------------------------------------------------------
time:   00:20:19
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053384/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053384/

    "2021-04-17T17:14:20.132388+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053385]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/workunits}
-----------------------------------------------------------------
time:   00:07:29
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053385/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053385/

    "2021-04-17T17:05:51.114431+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053387]  rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:03:43
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053387/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053387/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ebff6a1582594f548f6339063c102671

    Command failed on smithi016 with status 5: 'sudo systemctl stop
    ceph-c55297e8-9f9e-11eb-81bf-001a4aab830c@mon.a'

[6053390]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/set-chunks-read}
-----------------------------------------------------------------
time:   00:12:05
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053390/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053390/

    "2021-04-17T17:09:01.829247+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053391]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:20:36
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053391/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053391/

    "2021-04-17T17:18:55.761787+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053393]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:37:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053393/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053393/

    "2021-04-17T17:18:36.426467+0000 mgr.y (mgr.4118) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053397]  rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/test_rbd_api}
-----------------------------------------------------------------
time:   00:05:16
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053397/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053397/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bb2476899ade4e7c8edfcb8d6b03c587

    Command failed on smithi012 with status 5: 'sudo systemctl stop
    ceph-3135b606-9fa0-11eb-81bf-001a4aab830c@mon.a'

[6053400]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep}
-----------------------------------------------------------------
time:   00:23:01
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053400/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053400/

    "2021-04-17T17:16:16.647225+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053402]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced}
-----------------------------------------------------------------
time:   00:33:32
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053402/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053402/

    "2021-04-17T17:22:24.765388+0000 mgr.y (mgr.4122) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053403]  rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_18.04 workloads/radosbench_omap_write}
-----------------------------------------------------------------
time:   00:27:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053403/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053403/

    Found coredumps on ubuntu@smithi121.front.sepia.ceph.com

[6053404]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:09:27
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053404/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053404/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bef3ed8be881476db3f746639343b3d7

    Command failed on smithi051 with status 5: 'sudo systemctl stop ceph-
    cabfea12-9fa0-11eb-81bf-001a4aab830c@mon.a'

[6053405]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all}
-----------------------------------------------------------------
time:   00:18:14
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053405/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053405/

    "2021-04-17T17:17:04.353227+0000 mgr.x (mgr.4115) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053406]  rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:03:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053406/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053406/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fff950011a7241878589daaaac4c1368

    Command failed on smithi071 with status 5: 'sudo systemctl stop
    ceph-893c56f2-9fa0-11eb-81bf-001a4aab830c@mon.a'

[6053409]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects-localized}
-----------------------------------------------------------------
time:   00:30:03
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053409/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053409/

    "2021-04-17T17:25:23.557627+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053410]  rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 1-start 2-services/mirror 3-final}
-----------------------------------------------------------------
time:   00:03:46
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053410/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053410/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4924fb145a234e06b969bec0eca0326a

    Command failed on smithi016 with status 5: 'sudo systemctl stop
    ceph-9912820e-9fa0-11eb-81bf-001a4aab830c@mon.smithi016'

[6053412]  rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_orch_cli}
-----------------------------------------------------------------
time:   00:09:14
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053412/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053412/
sentry: https://sentry.ceph.com/organizations/ceph/?query=295d89d2aee94a498a80839023f31ae3

    Command failed on smithi163 with status 5: 'sudo systemctl stop
    ceph-d0132e10-9fa1-11eb-81bf-001a4aab830c@mon.a'

[6053414]  rados/upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm supported-random-distro$/{rhel_8} tasks upgrade/secondary}
-----------------------------------------------------------------
time:   00:16:48
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053414/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053414/
sentry: https://sentry.ceph.com/organizations/ceph/?query=00431ae8c86f451ab054a21de0fc804d

    An attempt to upgrade from a higher version to a lower one will always
    fail. Hint: check tags in the target git branch.

[6053415]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:36:03
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053415/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053415/

    "2021-04-17T17:31:01.817553+0000 mgr.x (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053417]  rados/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:05:43
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053417/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053417/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13dc837f7f87480fb43091f617278fc7

    Command failed on smithi036 with status 5: 'sudo systemctl stop
    ceph-b2a76e68-9fa1-11eb-81bf-001a4aab830c@mon.a'

[6053418]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects}
-----------------------------------------------------------------
time:   00:32:13
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053418/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053418/

    "2021-04-17T17:32:28.117194+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053419]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/crash}
-----------------------------------------------------------------
time:   00:09:47
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053419/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053419/

    "2021-04-17T17:28:54.782778+0000 mgr.z (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053421]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:20:34
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053421/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053421/

    "2021-04-17T17:39:25.780106+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053423]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:09:08
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053423/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053423/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0756ed6cfc244fb08e7efc4acddfc4a4

    Command failed on smithi058 with status 5: 'sudo systemctl stop
    ceph-2cad7e40-9fa3-11eb-81bf-001a4aab830c@mon.a'

[6053424]  rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites}
-----------------------------------------------------------------
time:   00:35:04
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053424/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053424/

    "2021-04-17T17:38:57.041120+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053425]  rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:39:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053425/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053425/

    "2021-04-17T17:42:30.511993+0000 mgr.x (mgr.5989) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053428]  rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews}
-----------------------------------------------------------------
time:   00:07:57
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053428/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053428/

    "2021-04-17T17:33:17.991973+0000 mgr.x (mgr.4116) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053429]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1}
-----------------------------------------------------------------
time:   00:24:35
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053429/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053429/

    "2021-04-17T17:34:11.341935+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053430]  rados/cephadm/smoke/{distro/centos_8.2_kubic_stable fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:06:35
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053430/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053430/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5dbfc28d603145b59335ae9f4a434e0b

    Command failed on smithi152 with status 5: 'sudo systemctl stop
    ceph-3e024ebe-9fa3-11eb-81bf-001a4aab830c@mon.a'

[6053431]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/rados_python}
-----------------------------------------------------------------
time:   00:17:15
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053431/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053431/

    "2021-04-17T17:34:55.363252+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053432]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced}
-----------------------------------------------------------------
time:   00:27:20
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053432/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053432/

    "2021-04-17T17:35:35.050502+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053434]  rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/cache-snaps}
-----------------------------------------------------------------
time:   00:05:06
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053434/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053434/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9bc18da8a6f44487869d6f9c36446c1e

    Command failed on smithi052 with status 5: 'sudo systemctl stop ceph-
    ac703d5c-9fa3-11eb-81bf-001a4aab830c@mon.a'

[6053437]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}
-----------------------------------------------------------------
time:   00:59:23
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053437/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053437/

    "2021-04-17T17:53:22.056342+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053442]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized}
-----------------------------------------------------------------
time:   00:26:36
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053442/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053442/

    "2021-04-17T17:49:18.457064+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053444]  rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/rgw 3-final}
-----------------------------------------------------------------
time:   00:03:58
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053444/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053444/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ad98d2cd6df04e1cbb16622e72e1db7e

    Command failed on smithi049 with status 5: 'sudo systemctl stop
    ceph-e81dbf22-9fa4-11eb-81bf-001a4aab830c@mon.smithi049'

[6053447]  rados/cephadm/smoke/{distro/rhel_8.3_kubic_stable fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:10:47
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053447/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053447/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bf0156f67c5e48e3b502ed2e13a45c70

    Command failed on smithi038 with status 5: 'sudo systemctl stop
    ceph-3b42c490-9fa5-11eb-81bf-001a4aab830c@mon.a'

[6053449]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:27:47
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053449/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053449/

    "2021-04-17T17:52:19.817443+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053452]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/failover}
-----------------------------------------------------------------
time:   00:11:32
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053452/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053452/

    "2021-04-17T17:51:16.987848+0000 mgr.y (mgr.4138) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053453]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:09:58
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053453/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053453/

    "2021-04-17T17:56:44.652623+0000 mgr.y (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053455]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch}
-----------------------------------------------------------------
time:   00:16:40
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053455/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053455/

    "2021-04-17T17:53:11.698137+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053456]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:08:56
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053456/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053456/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22753c47d123453f9e1964434d9fdae0

    Command failed on smithi090 with status 5: 'sudo systemctl stop
    ceph-6c0ecbb8-9fa6-11eb-81bf-001a4aab830c@mon.a'

[6053460]  rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:05:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053460/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053460/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5a25341717d74b9486082c029dfd0a57

    Command failed on smithi005 with status 5: 'sudo systemctl stop
    ceph-2c48c34e-9fa6-11eb-81bf-001a4aab830c@mon.a'

[6053462]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   03:03:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053462/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053462/
sentry: https://sentry.ceph.com/organizations/ceph/?query=71e1f79aaebf47f18120ae506511b920

    reached maximum tries (800) after waiting for 4800 seconds

[6053464]  rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:03:44
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053464/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053464/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d4b8e1c4ab57492cbd8e80cd5a431df0

    Command failed on smithi051 with status 5: 'sudo systemctl stop
    ceph-386679d2-9fa6-11eb-81bf-001a4aab830c@mon.a'

[6053466]  rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/radosbench}
-----------------------------------------------------------------
time:   00:05:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053466/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053466/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7978c5f6c08a4c558e233d53c18a7dfd

    Command failed on smithi156 with status 5: 'sudo systemctl stop ceph-
    de608a1c-9fa6-11eb-81bf-001a4aab830c@mon.a'

[6053469]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep}
-----------------------------------------------------------------
time:   00:16:33
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053469/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053469/

    "2021-04-17T18:06:37.297371+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053470]  rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/basic 3-final}
-----------------------------------------------------------------
time:   00:04:03
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053470/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053470/
sentry: https://sentry.ceph.com/organizations/ceph/?query=842c5f75c4424ff482b6e8f8059f32eb

    Command failed on smithi036 with status 5: 'sudo systemctl stop
    ceph-e12ad77a-9fa6-11eb-81bf-001a4aab830c@mon.smithi036'

[6053471]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests}
-----------------------------------------------------------------
time:   00:15:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053471/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053471/

    "2021-04-17T18:05:09.875486+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053474]  rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}
-----------------------------------------------------------------
time:   00:29:49
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053474/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053474/

    "2021-04-17T18:07:03.930904+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053480]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/rados_striper}
-----------------------------------------------------------------
time:   00:11:30
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053480/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053480/

    "2021-04-17T18:10:28.564747+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053482]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:08:38
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053482/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053482/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a1ee51af709f44818c7127fbc616ebd0

    Command failed on smithi109 with status 5: 'sudo systemctl stop
    ceph-61e2f75c-9fa8-11eb-81bf-001a4aab830c@mon.a'

[6053484]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:35:32
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053484/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053484/

    "2021-04-17T18:17:58.076009+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053485]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/insights}
-----------------------------------------------------------------
time:   00:14:59
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053485/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053485/

    "2021-04-17T18:16:32.753857+0000 mgr.z (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053486]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:12:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053486/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053486/

    "2021-04-17T18:19:25.994736+0000 mgr.y (mgr.4112) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053487]  rados/cephadm/with-work/{0-distro/centos_8.2_kubic_stable fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:08:49
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053487/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053487/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3e537928d33d41038dd38cda3a4db608

    Command failed on smithi086 with status 5: 'sudo systemctl stop ceph-
    eb8955a0-9fa8-11eb-81bf-001a4aab830c@mon.a'

[6053489]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-small}
-----------------------------------------------------------------
time:   00:12:36
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053489/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053489/

    "2021-04-17T18:13:44.756548+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053490]  rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:03:58
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053490/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053490/
sentry: https://sentry.ceph.com/organizations/ceph/?query=efd9a887907248ab97bdffc292fb871a

    Command failed on smithi051 with status 5: 'sudo systemctl stop
    ceph-52d23408-9fa8-11eb-81bf-001a4aab830c@mon.a'

[6053491]  rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:17:34
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053491/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053491/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e1267da50709463a87ecece3b37dd138

    Command failed (workunit test rados/version_number_sanity.sh) on smithi175
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a2e1e42f98caaf16ed5f3393512aaaefde169c3f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/rados/version_number_sanity.sh'

[6053495]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced}
-----------------------------------------------------------------
time:   00:21:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053495/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053495/

    "2021-04-17T18:20:29.248547+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053496]  rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_orch_cli}
-----------------------------------------------------------------
time:   00:08:51
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053496/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053496/
sentry: https://sentry.ceph.com/organizations/ceph/?query=56bc9461c43847e6adba88c6409c8346

    Command failed on smithi045 with status 5: 'sudo systemctl stop
    ceph-58cb297c-9fa9-11eb-81bf-001a4aab830c@mon.a'

[6053497]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy}
-----------------------------------------------------------------
time:   00:36:56
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053497/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053497/

    "2021-04-17T18:24:45.089257+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053498]  rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/rbd_cls}
-----------------------------------------------------------------
time:   00:05:40
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053498/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053498/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d75b828ab9a54622b41ca33f28bf80f0

    Command failed on smithi036 with status 5: 'sudo systemctl stop
    ceph-4d492c16-9fa9-11eb-81bf-001a4aab830c@mon.a'

[6053499]  rados/cephadm/smoke-roleless/{0-distro/centos_8.2_kubic_stable 1-start 2-services/iscsi 3-final}
-----------------------------------------------------------------
time:   00:06:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053499/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053499/
sentry: https://sentry.ceph.com/organizations/ceph/?query=10020e3cc5594c56b400428fd4c29707

    Command failed on smithi121 with status 5: 'sudo systemctl stop
    ceph-6734b73a-9fa9-11eb-81bf-001a4aab830c@mon.smithi121'

[6053501]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind}
-----------------------------------------------------------------
time:   00:29:39
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053501/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053501/

    "2021-04-17T18:36:39.161432+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053502]  rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{rhel_8} tasks/dashboard}
-----------------------------------------------------------------
time:   01:16:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053502/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053502/

    "2021-04-17T18:28:21.877201+0000 mgr.z (mgr.4120) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053508]  rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest}
-----------------------------------------------------------------
time:   00:21:31
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053508/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053508/

    "2021-04-17T18:39:09.619962+0000 mgr.y (mgr.4102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053509]  rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:16:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053509/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053509/
sentry: https://sentry.ceph.com/organizations/ceph/?query=adf4e553c7894623a6dbf72b13e41749

    Command failed (workunit test mon/test_mon_config_key.py) on smithi123 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a2e1e42f98caaf16ed5f3393512aaaefde169c3f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/mon/test_mon_config_key.py'

[6053510]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:09:18
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053510/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053510/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5ff20c74f80b4b709479dc1cc8fcbc74

    Command failed on smithi003 with status 5: 'sudo systemctl stop ceph-
    ed1fbb3c-9faa-11eb-81bf-001a4aab830c@mon.a'

[6053511]  rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:30:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053511/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053511/

    "2021-04-17T18:32:56.767512+0000 mgr.x (mgr.6837) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053512]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big}
-----------------------------------------------------------------
time:   00:33:47
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053512/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053512/

    "2021-04-17T18:28:29.272134+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053514]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps}
-----------------------------------------------------------------
time:   00:17:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053514/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053514/

    "2021-04-17T18:28:48.230641+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053515]  rados/cephadm/dashboard/{0-distro/centos_8.2_kubic_stable task/test_e2e}
-----------------------------------------------------------------
time:   00:08:51
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053515/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053515/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c62e2057ac9c408fa2148c52477a8425

    Command failed on smithi095 with status 5: 'sudo systemctl stop
    ceph-05ee82ba-9fab-11eb-81bf-001a4aab830c@mon.a'

[6053518]  rados/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_20.04} 2-node-mgr orchestrator_cli}
-----------------------------------------------------------------
time:   00:11:01
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053518/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053518/

    "2021-04-17T18:29:46.243929+0000 mgr.y (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053519]  rados/cephadm/smoke/{distro/centos_8.2_kubic_stable fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:06:30
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053519/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053519/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3227bce73f03466da3673aaf2e8d620e

    Command failed on smithi109 with status 5: 'sudo systemctl stop
    ceph-e084559a-9faa-11eb-81bf-001a4aab830c@mon.a'

[6053520]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:29:07
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053520/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053520/

    "2021-04-17T18:31:11.121516+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053521]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:10:23
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053521/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053521/

    "2021-04-17T18:38:06.681925+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053522]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced}
-----------------------------------------------------------------
time:   00:36:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053522/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053522/

    "2021-04-17T18:42:46.581350+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053523]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/module_selftest}
-----------------------------------------------------------------
time:   00:37:58
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053523/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053523/

    "2021-04-17T18:42:28.697816+0000 mgr.z (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053531]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-snaps}
-----------------------------------------------------------------
time:   00:20:54
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053531/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053531/

    "2021-04-17T18:43:47.689306+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053533]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:27:10
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053533/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053533/

    "2021-04-17T18:48:12.907729+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053536]  rados/cephadm/with-work/{0-distro/rhel_8.3_kubic_stable fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:18:44
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053536/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053536/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bb114769719746448aa10c527bbe3f9e

    Command failed on smithi064 with status 5: 'sudo systemctl stop
    ceph-1dce589e-9fae-11eb-81bf-001a4aab830c@mon.a'

[6053539]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix}
-----------------------------------------------------------------
time:   00:19:49
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053539/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053539/

    "2021-04-17T18:49:01.408850+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053540]  rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects}
-----------------------------------------------------------------
time:   00:05:38
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053540/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053540/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5911d254551b465a83e75d626b02dec2

    Command failed on smithi003 with status 5: 'sudo systemctl stop
    ceph-488de258-9fad-11eb-81bf-001a4aab830c@mon.a'

[6053542]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep}
-----------------------------------------------------------------
time:   00:24:13
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053542/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053542/

    "2021-04-17T18:51:05.380519+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053543]  rados/cephadm/smoke/{distro/rhel_8.3_kubic_stable fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:10:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053543/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053543/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fb1db274798d455982dd4b9bc1592da3

    Command failed on smithi095 with status 5: 'sudo systemctl stop
    ceph-787c07ba-9fad-11eb-81bf-001a4aab830c@mon.a'

[6053544]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache}
-----------------------------------------------------------------
time:   00:21:18
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053544/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053544/

    "2021-04-17T18:55:19.781775+0000 mgr.y (mgr.4098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053545]  rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/mirror 3-final}
-----------------------------------------------------------------
time:   00:10:21
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053545/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053545/
sentry: https://sentry.ceph.com/organizations/ceph/?query=93f2fb447aa8409c8b7f1f99f3a8815c

    Command failed on smithi152 with status 5: 'sudo systemctl stop
    ceph-9be9a770-9fad-11eb-81bf-001a4aab830c@mon.smithi152'

[6053550]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/dedup-io-mixed}
-----------------------------------------------------------------
time:   00:21:35
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053550/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053550/

    "2021-04-17T18:56:29.183286+0000 mgr.y (mgr.4113) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053551]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:09:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053551/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053551/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4030508de6704ae49099aa80e1f3510f

    Command failed on smithi025 with status 5: 'sudo systemctl stop
    ceph-35b17b76-9fae-11eb-81bf-001a4aab830c@mon.a'

[6053552]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:11:48
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053552/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053552/

    "2021-04-17T18:55:15.905440+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053555]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/progress}
-----------------------------------------------------------------
time:   00:26:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053555/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053555/

    "2021-04-17T19:00:36.334020+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053558]  rados/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:05:46
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053558/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053558/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9160bfd9fde64f07aaa2ad799019a921

    Command failed on smithi051 with status 5: 'sudo systemctl stop
    ceph-836a5068-9fae-11eb-81bf-001a4aab830c@mon.a'

[6053560]  rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:03:38
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053560/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053560/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd8d7da45feb4c18ba7fd3a9ba0ab33a

    Command failed on smithi078 with status 5: 'sudo systemctl stop
    ceph-68b69254-9fae-11eb-81bf-001a4aab830c@mon.a'

[6053562]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread}
-----------------------------------------------------------------
time:   00:31:43
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053562/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053562/

    "2021-04-17T18:58:14.056015+0000 mgr.x (mgr.4117) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053566]  rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews}
-----------------------------------------------------------------
time:   00:07:01
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053566/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053566/

    "2021-04-17T19:02:34.038400+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053567]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-many-deletes}
-----------------------------------------------------------------
time:   00:28:58
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053567/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053567/

    "2021-04-17T19:02:47.703811+0000 mgr.x (mgr.4101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053571]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/pool-snaps-few-objects}
-----------------------------------------------------------------
time:   00:32:43
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053571/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053571/

    "2021-04-17T19:11:15.621957+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053572]  rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/test_rbd_api}
-----------------------------------------------------------------
time:   00:06:02
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053572/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053572/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3dc2efdf23774641a29b59180c29803f

    Command failed on smithi095 with status 5: 'sudo systemctl stop
    ceph-f2d1ad2e-9faf-11eb-81bf-001a4aab830c@mon.a'

[6053575]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind}
-----------------------------------------------------------------
time:   03:30:32
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053575/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053575/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8771a41cd6494f1cbe58ab42b9a54f22

    Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi074 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a2e1e42f98caaf16ed5f3393512aaaefde169c3f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh'

[6053576]  rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites}
-----------------------------------------------------------------
time:   00:23:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053576/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053576/

    "2021-04-17T19:09:44.445748+0000 mgr.y (mgr.4098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053577]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:09:05
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053577/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053577/
sentry: https://sentry.ceph.com/organizations/ceph/?query=867dd9b4c24f4450866c9bcd8c677845

    Command failed on smithi064 with status 5: 'sudo systemctl stop
    ceph-b7be6ad2-9fb0-11eb-81bf-001a4aab830c@mon.a'

[6053580]  rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 1-start 2-services/rgw 3-final}
-----------------------------------------------------------------
time:   00:03:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053580/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053580/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5eee2758494e4f1f82ba4cf2055a7eb4

    Command failed on smithi106 with status 5: 'sudo systemctl stop
    ceph-3a104e70-9fb0-11eb-81bf-001a4aab830c@mon.smithi106'

[6053581]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/rados_api_tests}
-----------------------------------------------------------------
time:   00:29:08
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053581/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053581/

    "2021-04-17T19:12:11.121284+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053583]  rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:03:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053583/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053583/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1032d1969d004da78dfbe3495536d290

    Command failed on smithi144 with status 5: 'sudo systemctl stop
    ceph-7a765b6c-9fb0-11eb-81bf-001a4aab830c@mon.a'

[6053584]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:11:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053584/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053584/

    "2021-04-17T19:18:13.293780+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053586]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/prometheus}
-----------------------------------------------------------------
time:   00:10:49
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053586/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053586/

    "2021-04-17T19:15:34.815437+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053587]  rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_orch_cli}
-----------------------------------------------------------------
time:   00:09:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053587/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053587/
sentry: https://sentry.ceph.com/organizations/ceph/?query=27097bcd65564f22a96a1fa30b61a5bf

    Command failed on smithi194 with status 5: 'sudo systemctl stop
    ceph-54fe39ee-9fb1-11eb-81bf-001a4aab830c@mon.a'

[6053588]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/readwrite}
-----------------------------------------------------------------
time:   00:11:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053588/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053588/

    "2021-04-17T19:15:56.844732+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053589]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:22:56
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053589/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053589/

    "2021-04-17T19:15:43.027639+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053590]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency}
-----------------------------------------------------------------
time:   00:53:28
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053590/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053590/

    "2021-04-17T19:17:29.314164+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053592]  rados/upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm supported-random-distro$/{centos_8} tasks upgrade/primary}
-----------------------------------------------------------------
time:   00:07:22
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053592/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053592/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22682fded3e047998ecbb68f9d1c21a7

    An attempt to upgrade from a higher version to a lower one will always
    fail. Hint: check tags in the target git branch.

[6053594]  rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:06:01
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053594/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053594/
sentry: https://sentry.ceph.com/organizations/ceph/?query=90f36aec75684bec90e9cc262c99e2b9

    Command failed on smithi120 with status 5: 'sudo systemctl stop
    ceph-9795eacc-9fb1-11eb-81bf-001a4aab830c@mon.a'

[6053595]  rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:31:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053595/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053595/

    "2021-04-17T19:21:34.603388+0000 mgr.x (mgr.6281) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053598]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:08:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053598/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053598/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d840457d07a54f1b984850209816c4d5

    Command failed on smithi156 with status 5: 'sudo systemctl stop
    ceph-0d651674-9fb2-11eb-81bf-001a4aab830c@mon.a'

[6053601]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/radosbench}
-----------------------------------------------------------------
time:   01:15:14
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053601/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053601/

    "2021-04-17T19:21:49.253323+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053603]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:31:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053603/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053603/

    "2021-04-17T19:29:45.217850+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053605]  rados/cephadm/smoke/{distro/centos_8.2_kubic_stable fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:07:12
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053605/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053605/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1b41058a81114e5a8b45f1c0d11965de

    Command failed on smithi029 with status 5: 'sudo systemctl stop
    ceph-7eda8046-9fb2-11eb-81bf-001a4aab830c@mon.a'

[6053606]  rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/cache-snaps}
-----------------------------------------------------------------
time:   00:06:17
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053606/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053606/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7cbbab43ceca40f098cd9d4d7437365b

    Command failed on smithi058 with status 5: 'sudo systemctl stop ceph-
    db5a78e4-9fb2-11eb-81bf-001a4aab830c@mon.a'

[6053608]  rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 1-start 2-services/basic 3-final}
-----------------------------------------------------------------
time:   00:03:52
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053608/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053608/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e09b4ea71b0404c8a24a8bf306c3b86

    Command failed on smithi152 with status 5: 'sudo systemctl stop
    ceph-8cedd2d2-9fb2-11eb-81bf-001a4aab830c@mon.smithi152'

[6053610]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep}
-----------------------------------------------------------------
time:   00:10:36
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053610/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053610/

    "2021-04-17T19:29:52.962437+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053611]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect}
-----------------------------------------------------------------
time:   00:23:26
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053611/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053611/

    "2021-04-17T19:34:47.893150+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053612]  rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:19:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053612/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053612/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cd3a09dbac554fb6a4c35f91fe0d8ab0

    timed out waiting for admin_socket to appear after osd.0 restart

[6053616]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/repair_test}
-----------------------------------------------------------------
time:   00:18:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053616/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053616/

    "2021-04-17T19:31:55.639109+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053618]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:20:40
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053618/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053618/

    "2021-04-17T19:43:06.149726+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053620]  rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest}
-----------------------------------------------------------------
time:   00:19:20
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053620/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053620/

    "2021-04-17T19:45:56.695729+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053621]  rados/cephadm/smoke/{distro/rhel_8.3_kubic_stable fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:10:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053621/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053621/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f0d3afa84d744ba0aa726e6f85f8e85f

    Command failed on smithi037 with status 5: 'sudo systemctl stop
    ceph-4732d588-9fb4-11eb-81bf-001a4aab830c@mon.a'

[6053623]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests}
-----------------------------------------------------------------
time:   00:13:04
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053623/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053623/

    "2021-04-17T19:38:01.231608+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053624]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/workunits}
-----------------------------------------------------------------
time:   00:17:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053624/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053624/

    "2021-04-17T19:43:02.745003+0000 mgr.z (mgr.4114) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053626]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:40:13
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053626/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053626/

    "2021-04-17T19:44:54.911152+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053629]  rados/cephadm/with-work/{0-distro/centos_8.2_kubic_stable fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:09:02
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053629/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053629/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0322453c84ce44b7be1e21b5daaa38c2

    Command failed on smithi086 with status 5: 'sudo systemctl stop
    ceph-026e481e-9fb5-11eb-81bf-001a4aab830c@mon.a'

[6053631]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect_set_object}
-----------------------------------------------------------------
time:   00:11:53
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053631/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053631/

    "2021-04-17T19:42:30.967785+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053633]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:09:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053633/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053633/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f9092e1bc9c3424d85b7430e9cd1cde4

    Command failed on smithi121 with status 5: 'sudo systemctl stop
    ceph-203fb7f6-9fb5-11eb-81bf-001a4aab830c@mon.a'

[6053637]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:29:15
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053637/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053637/

    "2021-04-17T19:44:51.843468+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053639]  rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/iscsi 3-final}
-----------------------------------------------------------------
time:   00:04:12
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053639/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053639/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f36e79eef33648e5a455359cd16ec224

    Command failed on smithi003 with status 5: 'sudo systemctl stop
    ceph-4ec208cc-9fb5-11eb-81bf-001a4aab830c@mon.smithi003'

[6053641]  rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/radosbench}
-----------------------------------------------------------------
time:   00:05:20
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053641/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053641/
sentry: https://sentry.ceph.com/organizations/ceph/?query=238dfcbebbda40f7803ce8b53ba49412

    Command failed on smithi006 with status 5: 'sudo systemctl stop
    ceph-13c0d40a-9fb6-11eb-81bf-001a4aab830c@mon.a'

[6053642]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/set-chunks-read}
-----------------------------------------------------------------
time:   00:14:43
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053642/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053642/

    "2021-04-17T19:52:19.978599+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053645]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}
-----------------------------------------------------------------
time:   00:54:40
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053645/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053645/

    "2021-04-17T20:04:29.196356+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053646]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/scrub_test}
-----------------------------------------------------------------
time:   00:09:56
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053646/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053646/

    "2021-04-17T19:52:34.772232+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053647]  rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:03:51
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053647/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053647/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a31c659194b64b399d503b99ec87255b

    Command failed on smithi001 with status 5: 'sudo systemctl stop
    ceph-571fef38-9fb6-11eb-81bf-001a4aab830c@mon.a'

[6053651]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:12:04
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053651/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053651/

    "2021-04-17T20:01:35.025622+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053652]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects-balanced}
-----------------------------------------------------------------
time:   00:19:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053652/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053652/

    "2021-04-17T19:58:16.627075+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053654]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/crash}
-----------------------------------------------------------------
time:   00:19:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053654/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053654/

    "2021-04-17T20:03:50.905277+0000 mgr.z (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053656]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:09:20
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053656/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053656/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d9d52eb07a1d436291be53ad828bb40c

    Command failed on smithi086 with status 5: 'sudo systemctl stop ceph-
    ad6df8ac-9fb7-11eb-81bf-001a4aab830c@mon.a'

[6053660]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects-localized}
-----------------------------------------------------------------
time:   00:22:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053660/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053660/

    "2021-04-17T20:01:56.853670+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053661]  rados/cephadm/with-work/{0-distro/rhel_8.3_kubic_stable fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python}
-----------------------------------------------------------------
time:   00:18:24
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053661/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053661/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4fd1b6661304e6981e76288ef035472

    Command failed on smithi073 with status 5: 'sudo systemctl stop
    ceph-a2ad08f8-9fb8-11eb-81bf-001a4aab830c@mon.a'

[6053664]  rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:04:32
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053664/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053664/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4345bb9d2fd34ffd8f6272e4bb08bcba

    Command failed on smithi003 with status 5: 'sudo systemctl stop
    ceph-7f0c8fb4-9fb7-11eb-81bf-001a4aab830c@mon.a'

[6053665]  rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:08:02
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053665/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053665/
sentry: https://sentry.ceph.com/organizations/ceph/?query=556306f6ec4041e7bd025a0d0b8c9fb6

    Command failed (workunit test rados/version_number_sanity.sh) on smithi169
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a2e1e42f98caaf16ed5f3393512aaaefde169c3f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/rados/version_number_sanity.sh'

[6053668]  rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/libcephsqlite}
-----------------------------------------------------------------
time:   00:19:43
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053668/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053668/

    "2021-04-17T20:02:37.050257+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053670]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:30:43
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053670/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053670/

    "2021-04-17T20:04:21.206338+0000 mgr.x (mgr.4116) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053671]  rados/cephadm/smoke-roleless/{0-distro/centos_8.2_kubic_stable 1-start 2-services/mirror 3-final}
-----------------------------------------------------------------
time:   00:06:56
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053671/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053671/
sentry: https://sentry.ceph.com/organizations/ceph/?query=347ece11da744e0ea5babd0515b9fa87

    Command failed on smithi058 with status 5: 'sudo systemctl stop
    ceph-f1de3ae2-9fb7-11eb-81bf-001a4aab830c@mon.smithi058'

[6053674]  rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/rbd_cls}
-----------------------------------------------------------------
time:   00:05:19
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053674/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053674/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2fd334c2271844439e7c7b70394a12d3

    Command failed on smithi006 with status 5: 'sudo systemctl stop
    ceph-3130b9ae-9fb8-11eb-81bf-001a4aab830c@mon.a'

[6053675]  rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_orch_cli}
-----------------------------------------------------------------
time:   00:09:06
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053675/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053675/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9bc9489921194e09aa838808c1d9d302

    Command failed on smithi028 with status 5: 'sudo systemctl stop
    ceph-7103c31e-9fb8-11eb-81bf-001a4aab830c@mon.a'

[6053677]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_cls_all validater/lockdep}
-----------------------------------------------------------------
time:   00:16:36
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053677/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053677/

    "2021-04-17T20:11:05.199771+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053678]  rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:30:20
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053678/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053678/

    "2021-04-17T20:14:07.698140+0000 mgr.x (mgr.6318) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053679]  rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_8} tasks/dashboard}
-----------------------------------------------------------------
time:   01:09:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053679/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053679/

    "2021-04-17T20:11:26.412333+0000 mgr.y (mgr.4119) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053684]  rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:08:45
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053684/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053684/
sentry: https://sentry.ceph.com/organizations/ceph/?query=244ded4a823e474fbff4d5a5fff77efb

    Command failed on smithi095 with status 5: 'sudo systemctl stop
    ceph-b31503f2-9fb9-11eb-81bf-001a4aab830c@mon.a'

[6053685]  rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced}
-----------------------------------------------------------------
time:   00:28:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053685/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053685/

    "2021-04-17T20:17:06.182167+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053686]  rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
-----------------------------------------------------------------
time:   00:11:45
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053686/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053686/

    "2021-04-17T20:21:47.397718+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053687]  rados/cephadm/dashboard/{0-distro/centos_8.2_kubic_stable task/test_e2e}
-----------------------------------------------------------------
time:   00:08:56
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053687/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053687/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1485097f337e409aab625a1677c7189b

    Command failed on smithi086 with status 5: 'sudo systemctl stop
    ceph-5b5bad90-9fba-11eb-81bf-001a4aab830c@mon.a'

[6053689]  rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/failover}
-----------------------------------------------------------------
time:   00:13:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053689/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053689/

    "2021-04-17T20:21:01.524535+0000 mgr.y (mgr.4112) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053692]  rados/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_18.04} 2-node-mgr orchestrator_cli}
-----------------------------------------------------------------
time:   00:12:16
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053692/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053692/

    "2021-04-17T20:20:25.488969+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053693]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects-localized}
-----------------------------------------------------------------
time:   00:24:08
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053693/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053693/

    "2021-04-17T20:22:36.122269+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log

[6053694]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:28:18
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053694/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053694/

    "2021-04-17T20:23:09.461362+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: rook" in cluster log



Dead
=================================================================
[6053352]  rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.3-octopus 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/classic}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053352/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053352/

    hit max job timeout

[6053394]  rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/connectivity}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053394/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053394/

    hit max job timeout

[6053435]  rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/classic}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053435/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053435/

    hit max job timeout

[6053475]  rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.3-octopus 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/connectivity}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053475/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053475/

    hit max job timeout

[6053507]  rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{rhel_8.3_kubic_stable} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053507/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053507/

    hit max job timeout

[6053527]  rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/classic}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053527/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053527/

    hit max job timeout

[6053565]  rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/connectivity}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053565/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053565/

    hit max job timeout

[6053613]  rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.3-octopus 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/classic}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053613/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053613/

    hit max job timeout

[6053649]  rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/connectivity}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053649/
log:    http://qa-proxy.ceph.com/teuthology/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053649/

    hit max job timeout



Running
=================================================================
[6053682] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_20.04} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053682/



Pass
=================================================================
[6053344] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_8}}
time:   00:12:31
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053344/

[6053348] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:24
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053348/

[6053349] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}
time:   00:19:27
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053349/

[6053350] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/erasure-code}
time:   00:49:57
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053350/

[6053354] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_api_tests}
time:   00:25:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053354/

[6053355] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_adoption}
time:   00:10:01
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053355/

[6053356] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews}
time:   00:08:03
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053356/

[6053358] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_18.04 workloads/radosbench_4M_seq_read}
time:   00:09:29
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053358/

[6053359] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:10:23
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053359/

[6053360] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}}
time:   00:09:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053360/

[6053361] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053361/

[6053365] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:10:06
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053365/

[6053368] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}}
time:   00:08:21
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053368/

[6053369] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:16:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053369/

[6053370] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_cephadm}
time:   00:17:38
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053370/

[6053372] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mgr}
time:   00:20:07
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053372/

[6053376] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_8}}
time:   00:23:05
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053376/

[6053377] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}}
time:   00:08:44
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053377/

[6053379] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_18.04 workloads/radosbench_4M_write}
time:   00:08:40
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053379/

[6053380] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:15
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053380/

[6053386] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:33
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053386/

[6053388] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:27
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053388/

[6053389] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_mon_osdmap_prune}
time:   00:51:57
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053389/

[6053392] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery}
time:   00:09:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053392/

[6053395] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/misc}
time:   00:30:12
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053395/

[6053396] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_cephadm_repos}
time:   00:07:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053396/

[6053398] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}}
time:   00:18:57
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053398/

[6053399] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:24:34
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053399/

[6053401] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:09:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053401/

[6053407] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}}
time:   00:17:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053407/

[6053408] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}}
time:   01:57:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053408/

[6053411] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:48
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053411/

[6053413] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/mon}
time:   00:40:19
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053413/

[6053416] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}}
time:   00:45:23
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053416/

[6053420] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:08:06
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053420/

[6053422] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_18.04 workloads/sample_fio}
time:   00:12:18
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053422/

[6053426] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_mon_workunits}
time:   00:26:04
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053426/

[6053427] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/none msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053427/

[6053433] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:18:51
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053433/

[6053436] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:09:53
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053436/

[6053438] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}}
time:   01:05:27
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053438/

[6053439] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/osd-backfill}
time:   01:40:23
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053439/

[6053440] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_adoption}
time:   00:10:12
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053440/

[6053441] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{rhel_8}}
time:   02:19:36
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053441/

[6053443] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:14:12
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053443/

[6053445] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_18.04 workloads/sample_radosbench}
time:   00:09:29
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053445/

[6053446] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
time:   00:45:39
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053446/

[6053448] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects}
time:   00:23:35
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053448/

[6053450] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_cephadm}
time:   00:16:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053450/

[6053451] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:31:10
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053451/

[6053454] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:13
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053454/

[6053457] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/osd}
time:   01:51:17
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053457/

[6053458] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/many workloads/snaps-few-objects}
time:   00:40:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053458/

[6053459] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed}
time:   00:13:26
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053459/

[6053461] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews}
time:   00:08:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053461/

[6053463] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:16:18
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053463/

[6053465] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}
time:   00:10:14
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053465/

[6053467] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_18.04 workloads/cosbench_64K_read_write}
time:   00:16:42
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053467/

[6053468] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:32:23
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053468/

[6053472] rados/objectstore/{backends/objectstore-filestore-memstore supported-random-distro$/{ubuntu_latest}}
time:   00:28:18
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053472/

[6053473] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:45
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053473/

[6053476] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}
time:   00:10:37
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053476/

[6053477] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_cephadm_repos}
time:   00:07:17
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053477/

[6053478] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-big}
time:   00:30:57
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053478/

[6053479] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/scrub}
time:   01:41:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053479/

[6053481] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:28:58
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053481/

[6053483] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:19
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053483/

[6053488] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_18.04 workloads/cosbench_64K_write}
time:   00:27:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053488/

[6053492] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}
time:   00:11:36
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053492/

[6053493] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/one workloads/pool-create-delete}
time:   00:26:47
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053493/

[6053494] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery}
time:   00:22:54
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053494/

[6053500] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:12:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053500/

[6053503] rados/objectstore/{backends/alloc-hint supported-random-distro$/{ubuntu_latest}}
time:   00:06:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053503/

[6053504] rados/rest/{mgr-restful supported-random-distro$/{rhel_8}}
time:   00:18:30
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053504/

[6053505] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:12:18
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053505/

[6053506] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/crush}
time:   00:12:38
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053506/

[6053513] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_18.04 workloads/fio_4K_rand_read}
time:   00:11:08
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053513/

[6053516] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:48
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053516/

[6053517] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053517/

[6053524] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}}
time:   00:21:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053524/

[6053525] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{rhel_8}}
time:   00:16:59
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053525/

[6053526] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites}
time:   00:22:20
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053526/

[6053528] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/erasure-code}
time:   00:42:15
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053528/

[6053529] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_adoption}
time:   00:09:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053529/

[6053530] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_5925}
time:   00:11:03
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053530/

[6053532] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews}
time:   00:06:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053532/

[6053534] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_18.04 workloads/fio_4K_rand_rw}
time:   00:13:23
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053534/

[6053535] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
time:   00:18:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053535/

[6053537] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:16:58
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053537/

[6053538] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}}
time:   00:10:29
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053538/

[6053541] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:36:14
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053541/

[6053546] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}}
time:   00:23:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053546/

[6053547] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:07:20
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053547/

[6053548] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/mgr}
time:   00:11:13
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053548/

[6053549] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_cephadm}
time:   00:17:02
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053549/

[6053553] rados/singleton/{all/peer mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}}
time:   00:16:55
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053553/

[6053554] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_18.04 workloads/fio_4M_rand_read}
time:   00:09:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053554/

[6053556] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:21:46
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053556/

[6053557] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:08:42
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053557/

[6053559] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps}
time:   00:11:19
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053559/

[6053561] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}}
time:   00:19:03
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053561/

[6053563] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_api_tests}
time:   00:36:03
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053563/

[6053564] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{rhel_8}}
time:   00:16:28
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053564/

[6053568] rados/objectstore/{backends/filejournal supported-random-distro$/{rhel_8}}
time:   00:17:14
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053568/

[6053569] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/misc}
time:   00:38:38
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053569/

[6053570] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_cephadm_repos}
time:   00:06:52
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053570/

[6053573] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}}
time:   00:09:16
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053573/

[6053574] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:09:30
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053574/

[6053578] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:18:38
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053578/

[6053579] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_18.04 workloads/fio_4M_rand_rw}
time:   00:09:33
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053579/

[6053582] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}}
time:   00:16:35
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053582/

[6053585] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{rhel_8}}
time:   00:17:13
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053585/

[6053591] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/mon}
time:   00:41:42
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053591/

[6053593] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}}
time:   00:13:54
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053593/

[6053596] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:19:22
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053596/

[6053597] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_18.04 workloads/fio_4M_rand_write}
time:   00:09:47
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053597/

[6053599] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{rhel_8}}
time:   02:24:22
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053599/

[6053600] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_osdmap_prune}
time:   00:39:57
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053600/

[6053602] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_recovery}
time:   00:22:22
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053602/

[6053604] rados/singleton/{all/random-eio mon_election/classic msgr-failures/none msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}}
time:   00:30:34
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053604/

[6053607] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{rhel_8}}
time:   00:18:28
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053607/

[6053609] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:26:22
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053609/

[6053614] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/osd-backfill}
time:   01:47:00
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053614/

[6053615] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_adoption}
time:   00:11:20
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053615/

[6053617] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:14:23
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053617/

[6053619] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_18.04 workloads/radosbench_4K_rand_read}
time:   00:11:30
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053619/

[6053622] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
time:   00:15:58
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053622/

[6053625] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites}
time:   00:32:19
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053625/

[6053627] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_cephadm}
time:   00:16:38
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053627/

[6053628] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:23:51
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053628/

[6053630] rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}}
time:   00:08:42
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053630/

[6053632] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{ubuntu_latest}}
time:   02:18:16
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053632/

[6053634] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_mon_workunits}
time:   00:37:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053634/

[6053635] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/osd}
time:   02:10:05
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053635/

[6053636] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews}
time:   00:07:47
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053636/

[6053638] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:18:15
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053638/

[6053640] rados/singleton/{all/test-crash mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}
time:   00:08:30
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053640/

[6053643] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_18.04 workloads/radosbench_4K_seq_read}
time:   00:10:24
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053643/

[6053644] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:10:40
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053644/

[6053648] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:05
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053648/

[6053650] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}}
time:   00:14:54
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053650/

[6053653] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_cephadm_repos}
time:   00:07:10
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053653/

[6053655] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/scrub}
time:   01:42:27
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053655/

[6053657] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:28:57
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053657/

[6053658] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:35:32
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053658/

[6053659] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}}
time:   01:16:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053659/

[6053662] rados/objectstore/{backends/fusestore supported-random-distro$/{rhel_8}}
time:   00:16:57
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053662/

[6053663] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_18.04 workloads/radosbench_4M_rand_read}
time:   00:10:58
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053663/

[6053666] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}
time:   00:36:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053666/

[6053667] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects}
time:   00:30:25
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053667/

[6053669] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews}
time:   00:06:34
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053669/

[6053672] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/small-objects}
time:   00:32:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053672/

[6053673] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}
time:   00:38:43
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053673/

[6053676] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:27:34
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053676/

[6053680] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:13:50
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053680/

[6053681] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/crush}
time:   00:20:16
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053681/

[6053683] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}}
time:   00:27:56
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053683/

[6053688] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_18.04 workloads/radosbench_4M_seq_read}
time:   00:09:41
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053688/

[6053690] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}}
time:   00:19:09
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053690/

[6053691] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:09:11
info:   http://pulpito.ceph.com/kchai-2021-04-17_15:15:57-rados-wip-kefu-testing-2021-04-16-2148-distro-basic-smithi/6053691/