2021-08-17T04:50:41.790 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-17T04:50:41.792 INFO:teuthology.results:subset: '101200/99999' 2021-08-17T04:50:41.793 INFO:teuthology.results:seed: '2477' 2021-08-17T08:03:04.961 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-17T08:03:04.998 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-08-17T20:03:12.467 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-08-17T20:04:01.415 INFO:scrape:Found 367 jobs 2021-08-17T20:04:10.321 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.322 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:10.336 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:10.348 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.349 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:10.360 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.362 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:10.378 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.379 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:10.392 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.393 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:10.405 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.406 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:10.418 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:10.430 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:10.442 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7031-g8b589883 (8b5898830ff89e9ef894d372c8802cccce043e9d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f9660daf3c0] , 2: gsignal() 2021-08-17T20:04:10.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-08-17T06:39:16.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-08-17T06:39:16.676 INFO:teuthology.task.internal:Transferring archived files... , 2021-08-17T06:39:16.678 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/remote/smithi022 2021-08-17T20:04:13.586 INFO:scrape:Found 12 distinct failure reasons 2021-08-17T20:04:13.587 INFO:scrape:Failure: Command failed on smithi151 with status 5: 'sudo systemctl stop ceph-df39385e-ff18-11eb-8c24-001a4aab830c@mon.smithi151' 2021-08-17T20:04:13.587 INFO:scrape:47 jobs 2021-08-17T20:04:13.588 INFO:scrape:suites intersection: [] 2021-08-17T20:04:13.589 INFO:scrape:suites union: ['1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-bootstrap/octopus', '1-start', '2-nfs', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/mirror', '2-services/nfs', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw', '2-services/nfs-ingress2', '2-services/nfs2', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-with-workload', '4-final}', 'debug/mgr', 'fixed-2', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_kubic_stable', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable', 'rados/cephadm/smoke-singlehost/{0-distro$/{centos_8.3_kubic_stable}', 'rados/cephadm/smoke/{distro/centos_8.3_kubic_stable', 'rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable', 'rados/cephadm/with-work/{0-distro/centos_8.3_kubic_stable', 'rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable', 'rados/dashboard/{centos_8.2_kubic_stable', 'random-objectstore$/{bluestore-comp-zlib}', 'random-objectstore$/{filestore-xfs}', 'root}', 'start', 'start}', 'task/test_nfs}', 'task/test_orch_cli}', 'tasks/e2e}', 'tasks/rados_api_tests}'] 2021-08-17T20:04:13.589 INFO:scrape: 2021-08-17T20:04:13.590 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi001 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2021-08-17T20:04:13.591 INFO:scrape:4 jobs: ['6343616', '6343702', '6343515', '6343430'] 2021-08-17T20:04:13.592 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable', 'task/test_cephadm}'] 2021-08-17T20:04:13.592 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable', 'task/test_cephadm}'] 2021-08-17T20:04:13.593 INFO:scrape: 2021-08-17T20:04:13.593 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rgw.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' 2021-08-17T20:04:13.593 INFO:scrape:3 jobs: ['6343580', '6343494', '6343639'] 2021-08-17T20:04:13.594 INFO:scrape:suites intersection: ['mon_election/classic'] 2021-08-17T20:04:13.595 INFO:scrape:suites union: ['0-start', '1-tasks', 'ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/upgrade/parallel/{0-distro$/{rhel_8.3_kubic_stable}', 'rados/verify/{centos_latest', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'tasks/rados_cls_all', 'test_rbd_api', 'test_rbd_python}}', 'thrashosds-health}', 'upgrade-sequence', 'validater/valgrind}', 'workload/{ec-rados-default'] 2021-08-17T20:04:13.595 INFO:scrape: 2021-08-17T20:04:13.596 INFO:scrape:Failure: {'Failure object was': {'smithi007.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} 2021-08-17T20:04:13.596 INFO:scrape:3 jobs: ['6343408', '6343734', '6343435'] 2021-08-17T20:04:13.597 INFO:scrape:suites intersection: [] 2021-08-17T20:04:13.597 INFO:scrape:suites union: ['clusters/9-mons', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/bluestore-low-osd-mem-target', 'openstack', 'rados', 'rados/monthrash/{ceph', 'rados/perf/{ceph', 'rados/singleton-nomsgr/{all/ceph-post-file', 'scheduler/dmclock_1Shard_16Threads', 'settings/optimized', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/many', 'ubuntu_latest', 'workloads/fio_4M_rand_read}', 'workloads/snaps-few-objects}'] 2021-08-17T20:04:13.598 INFO:scrape: 2021-08-17T20:04:13.598 INFO:scrape:Dead: b'2021-08-17T17:56:31.250 DEBUG:teuthology.exit:Finished running handlers' 2021-08-17T20:04:13.599 INFO:scrape:3 jobs: ['6343512', '6343649', '6343463'] 2021-08-17T20:04:13.599 INFO:scrape:suites intersection: ['rados'] 2021-08-17T20:04:13.600 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/singleton/{all/thrash-eio', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/snaps-few-objects}'] 2021-08-17T20:04:13.600 INFO:scrape: 2021-08-17T20:04:13.601 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2021-08-17T20:04:13.602 INFO:scrape:1 jobs: ['6343711'] 2021-08-17T20:04:13.602 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}'] 2021-08-17T20:04:13.603 INFO:scrape: 2021-08-17T20:04:13.603 INFO:scrape:Failure: 'wait for operator' reached maximum tries (90) after waiting for 900 seconds 2021-08-17T20:04:13.604 INFO:scrape:2 jobs: ['6343389', '6343665'] 2021-08-17T20:04:13.604 INFO:scrape:suites intersection: ['0-kubeadm', '1-rook', '2-workload/radosbench', '3-final', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'rados/rook/smoke/{0-distro/ubuntu_20.04'] 2021-08-17T20:04:13.605 INFO:scrape:suites union: ['0-kubeadm', '1-rook', '2-workload/radosbench', '3-final', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.0}', 'rook/master}'] 2021-08-17T20:04:13.605 INFO:scrape: 2021-08-17T20:04:13.606 INFO:scrape:Failure: "2021-08-17T05:26:45.112989+0000 mon.a (mon.0) 17 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log 2021-08-17T20:04:13.606 INFO:scrape:2 jobs: ['6343424', '6343576'] 2021-08-17T20:04:13.606 INFO:scrape:suites intersection: [] 2021-08-17T20:04:13.607 INFO:scrape:suites union: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/fastclose', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/objectstore/{backends/alloc-hint', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2021-08-17T20:04:13.607 INFO:scrape: 2021-08-17T20:04:13.608 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2021-08-17T20:04:13.608 INFO:scrape:4 jobs: ['6343505', '6343603', '6343689', '6343418'] 2021-08-17T20:04:13.609 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable', 'task/test_adoption}'] 2021-08-17T20:04:13.610 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable', 'task/test_adoption}'] 2021-08-17T20:04:13.610 INFO:scrape: 2021-08-17T20:04:13.611 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2021-08-17T20:04:13.611 INFO:scrape:1 jobs: ['6343440'] 2021-08-17T20:04:13.612 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_8}', 'tasks/rados_api_tests}'] 2021-08-17T20:04:13.613 INFO:scrape: 2021-08-17T20:04:13.613 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*SyntheticMatrixC*/2 --gtest_catch_exceptions=0\'' 2021-08-17T20:04:13.614 INFO:scrape:1 jobs: ['6343511'] 2021-08-17T20:04:13.614 INFO:scrape:suites: ['rados/objectstore/{backends/objectstore-bluestore-b', 'supported-random-distro$/{ubuntu_latest}}'] 2021-08-17T20:04:13.614 INFO:scrape: 2021-08-17T20:04:13.615 INFO:scrape:Dead: b'2021-08-17T20:04:12.454 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops' 2021-08-17T20:04:13.615 INFO:scrape:1 jobs: ['6343746'] 2021-08-17T20:04:13.616 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] 2021-08-17T20:04:13.616 INFO:scrape: 2021-08-17T20:04:13.629 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/ failed: 65 dead: 6 running: 1 waiting: 0 queued: 0 passed: 295 Fail ================================================================= [6343383] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/nfs-ingress-rgw 3-final} ----------------------------------------------------------------- time: 00:06:32 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343383/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f892a140105645c2af623998ccb5c5d4 Command failed on smithi025 with status 5: 'sudo systemctl stop ceph- dcd09828-ff18-11eb-8c24-001a4aab830c@mon.smithi025' [6343388] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:07:46 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343388/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=25086d8e279f4abdb52a5ff0d600007c Command failed on smithi125 with status 5: 'sudo systemctl stop ceph-b4520634-ff18-11eb-8c24-001a4aab830c@mon.a' [6343389] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 1-rook 2-workload/radosbench 3-final cluster/3-node k8s/1.21 net/calico rook/1.7.0} ----------------------------------------------------------------- time: 00:21:00 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343389/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343389/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89eb0f1ae8814a9fadc6bfe01c4ed9f6 'wait for operator' reached maximum tries (90) after waiting for 900 seconds [6343396] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:08:13 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343396/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343396/ sentry: https://sentry.ceph.com/organizations/ceph/?query=825f3effcb4641348e65ba55e72dc2b0 Command failed on smithi028 with status 5: 'sudo systemctl stop ceph-2f116194-ff19-11eb-8c24-001a4aab830c@mon.a' [6343398] rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_kubic_stable 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:05:22 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343398/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343398/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc16d555b2ca4b4a85c0b6398a0003f4 Command failed on smithi151 with status 5: 'sudo systemctl stop ceph- df39385e-ff18-11eb-8c24-001a4aab830c@mon.smithi151' [6343404] rados/cephadm/smoke/{distro/centos_8.3_kubic_stable fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:06:38 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343404/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ae2425b6ff64a9b8ba7dd41f5264fdf Command failed on smithi023 with status 5: 'sudo systemctl stop ceph- cb7ae316-ff19-11eb-8c24-001a4aab830c@mon.a' [6343418] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:09:01 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343418/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343418/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ed4365c3e9548d78a0f44acb93d28d1 Command failed (workunit test cephadm/test_adoption.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [6343424] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:10:15 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343424/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343424/ "2021-08-17T05:26:45.112989+0000 mon.a (mon.0) 17 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [6343426] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:06:43 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343426/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ceddf8e1b7044d65920657bb200aafa5 Command failed on smithi025 with status 5: 'sudo systemctl stop ceph-4b639a40-ff1b-11eb-8c24-001a4aab830c@mon.smithi025' [6343430] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:08:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343430/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343430/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e762cda3119f42d984a0278a1a3fa607 Command failed (workunit test cephadm/test_cephadm.sh) on smithi039 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [6343434] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:08:22 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343434/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343434/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a54c2aa7e5c4ae3bc5af28144b3b9c4 Command failed on smithi150 with status 5: 'sudo systemctl stop ceph-93bad114-ff1b-11eb-8c24-001a4aab830c@mon.a' [6343440] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:25:14 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343440/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343440/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f67919449d12440bbb0f02196bb08ad0 Command failed (workunit test rados/test.sh) on smithi023 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [6343450] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:06:32 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343450/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343450/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f768e75437d4da69f3fa1c7705d58c4 Command failed on smithi142 with status 5: 'sudo systemctl stop ceph-745b34d4-ff1c-11eb-8c24-001a4aab830c@mon.smithi142' [6343462] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:07:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343462/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=349f3f03f2a6416bb82968d31108b9f9 Command failed on smithi115 with status 5: 'sudo systemctl stop ceph-26a02262-ff1d-11eb-8c24-001a4aab830c@mon.a' [6343465] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:08:12 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343465/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d9a530a2b7a4f0d9ba4e7c81421a9be Command failed on smithi039 with status 5: 'sudo systemctl stop ceph-88ae3d54-ff1d-11eb-8c24-001a4aab830c@mon.a' [6343471] rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_kubic_stable 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:05:31 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343471/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f321dd01e009450f9cdd08947e5f7f17 Command failed on smithi067 with status 5: 'sudo systemctl stop ceph-8b6597b8-ff1d-11eb-8c24-001a4aab830c@mon.smithi067' [6343482] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:07:49 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343482/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343482/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07fb10e9c2c24546ad956b6f4a16e908 Command failed on smithi192 with status 5: 'sudo systemctl stop ceph-887c1e54-ff1e-11eb-8c24-001a4aab830c@mon.a' [6343486] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:06:29 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343486/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343486/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d46cb7ff31c14319ad42d741de25ab0c Command failed on smithi064 with status 5: 'sudo systemctl stop ceph-4ecd4e5c-ff1f-11eb-8c24-001a4aab830c@mon.smithi064' [6343487] rados/dashboard/{centos_8.2_kubic_stable debug/mgr mon_election/classic random-objectstore$/{filestore-xfs} tasks/e2e} ----------------------------------------------------------------- time: 00:08:12 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343487/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343487/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b21ace3b9dd4484e8ab18427093d9863 Command failed on smithi016 with status 5: 'sudo systemctl stop ceph-13ec9a68-ff1f-11eb-8c24-001a4aab830c@mon.a' [6343491] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:08:03 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343491/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343491/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5959ed0acd7540b9a506093c40c68114 Command failed on smithi103 with status 5: 'sudo systemctl stop ceph-4cf65650-ff1f-11eb-8c24-001a4aab830c@mon.a' [6343494] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 04:38:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343494/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343494/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4de84f0158284514bab3c0a1c9e1dcc6 Command failed (workunit test cls/test_cls_rgw.sh) on smithi003 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6343496] rados/cephadm/smoke/{distro/centos_8.3_kubic_stable fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:06:19 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343496/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343496/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f8640f223a944afb81c2c2bf23fa90e Command failed on smithi151 with status 5: 'sudo systemctl stop ceph-f4d24c3a-ff1f-11eb-8c24-001a4aab830c@mon.a' [6343505] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_adoption} ----------------------------------------------------------------- time: 00:08:46 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343505/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343505/ sentry: https://sentry.ceph.com/organizations/ceph/?query=609a6e4eb8f646c6b7b93649f83cd217 Command failed (workunit test cephadm/test_adoption.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [6343511] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:40:56 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343511/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38a01feb16524f34946216d6724ff190 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*SyntheticMatrixC*/2 --gtest_catch_exceptions=0\'' [6343515] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:08:58 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343515/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343515/ sentry: https://sentry.ceph.com/organizations/ceph/?query=561c525c55b846a498019243f6e24f4d Command failed (workunit test cephadm/test_cephadm.sh) on smithi059 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [6343520] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:06:35 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343520/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343520/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3afdaa0597d4512b99bf4f20398daae Command failed on smithi057 with status 5: 'sudo systemctl stop ceph- ad205aa6-ff21-11eb-8c24-001a4aab830c@mon.smithi057' [6343525] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:07:58 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343525/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343525/ sentry: https://sentry.ceph.com/organizations/ceph/?query=516a85c85e6648b8bb393a716aa3cd80 Command failed on smithi109 with status 5: 'sudo systemctl stop ceph-9480e312-ff21-11eb-8c24-001a4aab830c@mon.a' [6343527] rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_kubic_stable 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:05:32 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343527/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343527/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c7b98b75e6da4854a56e50d773b66ced Command failed on smithi105 with status 5: 'sudo systemctl stop ceph-7954371a-ff21-11eb-8c24-001a4aab830c@mon.smithi105' [6343546] rados/cephadm/with-work/{0-distro/centos_8.3_kubic_stable fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:09:12 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343546/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343546/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03160474edaa482b889af7558eebde76 Command failed on smithi039 with status 5: 'sudo systemctl stop ceph-a62a5830-ff23-11eb-8c24-001a4aab830c@mon.a' [6343548] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:06:39 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343548/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343548/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43c9fe90dad2448e8cad50ab42f566d5 Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-6efaa39c-ff23-11eb-8c24-001a4aab830c@mon.smithi043' [6343551] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:07:52 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343551/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343551/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2d1ff7287f3437d93f84e28c1773283 Command failed on smithi197 with status 5: 'sudo systemctl stop ceph-6989d57c-ff23-11eb-8c24-001a4aab830c@mon.a' [6343556] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:08:10 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343556/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343556/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f2af230017547afb4f072cfb449d78b Command failed on smithi109 with status 5: 'sudo systemctl stop ceph-a73fcc1e-ff23-11eb-8c24-001a4aab830c@mon.a' [6343567] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:08:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343567/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343567/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9aceb22ba4347b48ed2e791c5526236 Command failed on smithi064 with status 5: 'sudo systemctl stop ceph-14893152-ff24-11eb-8c24-001a4aab830c@mon.a' [6343574] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:06:33 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343574/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343574/ sentry: https://sentry.ceph.com/organizations/ceph/?query=092fc1126606442ea0e044af44c1c471 Command failed on smithi051 with status 5: 'sudo systemctl stop ceph-222254b4-ff25-11eb-8c24-001a4aab830c@mon.smithi051' [6343576] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:09:21 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343576/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343576/ "2021-08-17T06:35:54.522087+0000 mon.a (mon.0) 121 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6343580] rados/upgrade/parallel/{0-distro$/{rhel_8.3_kubic_stable} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:46:28 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343580/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343580/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ed13c5155c54833952c1991b1b8dc68 Command failed (workunit test cls/test_cls_rgw.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6343582] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:08:18 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343582/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343582/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba54b4ee39e34c5ebceca1df2b29aa70 Command failed on smithi067 with status 5: 'sudo systemctl stop ceph-9dca4c70-ff25-11eb-8c24-001a4aab830c@mon.a' [6343586] rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_kubic_stable 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:05:32 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343586/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343586/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed6ff045587847919d6ebe5967f7d511 Command failed on smithi109 with status 5: 'sudo systemctl stop ceph-67e856d8-ff25-11eb-8c24-001a4aab830c@mon.smithi109' [6343592] rados/cephadm/smoke/{distro/centos_8.3_kubic_stable fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:06:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343592/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343592/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8d8aadede20473daf24ad1307cd8c65 Command failed on smithi131 with status 5: 'sudo systemctl stop ceph-0d98bd2a-ff26-11eb-8c24-001a4aab830c@mon.a' [6343597] rados/cephadm/smoke-singlehost/{0-distro$/{centos_8.3_kubic_stable} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:06:22 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343597/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343597/ sentry: https://sentry.ceph.com/organizations/ceph/?query=458ff4ca9a044cbda3c4531e746b86b9 Command failed on smithi190 with status 5: 'sudo systemctl stop ceph-5b52f9ae-ff26-11eb-8c24-001a4aab830c@mon.smithi190' [6343603] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:09:16 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343603/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343603/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28bcc7fbad6545538b24d9a36a1d128f Command failed (workunit test cephadm/test_adoption.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [6343616] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:09:05 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343616/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343616/ sentry: https://sentry.ceph.com/organizations/ceph/?query=36ea0d24c1b44747ae4837e6745f9234 Command failed (workunit test cephadm/test_cephadm.sh) on smithi001 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [6343618] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:08:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343618/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343618/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5cdbc764dbfd4b44bf1e4465677d9ad0 Command failed on smithi071 with status 5: 'sudo systemctl stop ceph-e4d4ed44-ff27-11eb-8c24-001a4aab830c@mon.a' [6343623] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:06:47 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343623/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343623/ sentry: https://sentry.ceph.com/organizations/ceph/?query=013ca090e54b4b1fb664261110e6a74f Command failed on smithi032 with status 5: 'sudo systemctl stop ceph-46998b16-ff28-11eb-8c24-001a4aab830c@mon.smithi032' [6343639] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 04:40:38 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343639/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343639/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bee24bb2174f452ba2e19cf85e69bec2 Command failed (workunit test cls/test_cls_rgw.sh) on smithi153 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6343647] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:08:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343647/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343647/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c21bbec1a4f474198ea28bb4ddd1517 Command failed on smithi001 with status 5: 'sudo systemctl stop ceph-d21f674a-ff29-11eb-8c24-001a4aab830c@mon.a' [6343648] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:08:05 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343648/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343648/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9583ee43d85443d3bd0f64418a2c1856 Command failed on smithi071 with status 5: 'sudo systemctl stop ceph- fdccd92c-ff29-11eb-8c24-001a4aab830c@mon.a' [6343652] rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_kubic_stable 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:05:26 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343652/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343652/ sentry: https://sentry.ceph.com/organizations/ceph/?query=824a2341a17445b7805e657f7768cbda Command failed on smithi097 with status 5: 'sudo systemctl stop ceph- edc54096-ff29-11eb-8c24-001a4aab830c@mon.smithi097' [6343657] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:06:26 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343657/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343657/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d6ffbe94f224dfcacc9b491897a5dc5 Command failed on smithi039 with status 5: 'sudo systemctl stop ceph- ace00d4e-ff2a-11eb-8c24-001a4aab830c@mon.smithi039' [6343661] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:08:22 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343661/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343661/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cff5f1d13f48482a9dbcefe266c4c6a0 Command failed on smithi008 with status 5: 'sudo systemctl stop ceph-9d0b9050-ff2a-11eb-8c24-001a4aab830c@mon.a' [6343665] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 1-rook 2-workload/radosbench 3-final cluster/3-node k8s/1.21 net/calico rook/master} ----------------------------------------------------------------- time: 00:30:40 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343665/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343665/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5c1c7b722824abb8fe31e04e044714d 'wait for teardown' reached maximum tries (100) after waiting for 500 seconds [6343671] rados/dashboard/{centos_8.2_kubic_stable debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} tasks/e2e} ----------------------------------------------------------------- time: 00:08:09 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343671/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343671/ sentry: https://sentry.ceph.com/organizations/ceph/?query=558f3b0ce0e64bffb8dbdaaeaac2fe37 Command failed on smithi064 with status 5: 'sudo systemctl stop ceph-b8494f1e-ff2b-11eb-8c24-001a4aab830c@mon.a' [6343676] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:07:54 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343676/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343676/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29ff028f950e4af5b3c07ebef9a264c0 Command failed on smithi097 with status 5: 'sudo systemctl stop ceph-0a28db56-ff2c-11eb-8c24-001a4aab830c@mon.a' [6343680] rados/cephadm/smoke/{distro/centos_8.3_kubic_stable fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:06:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343680/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343680/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7639eeb9590f4e6ab731f290b9eca202 Command failed on smithi036 with status 5: 'sudo systemctl stop ceph-d32e47a2-ff2c-11eb-8c24-001a4aab830c@mon.a' [6343683] rados/cephadm/smoke-singlehost/{0-distro$/{centos_8.3_kubic_stable} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:06:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343683/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343683/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ff6c37c57a04fd9bce1c0e47b93fa32 Command failed on smithi017 with status 5: 'sudo systemctl stop ceph-b751b988-ff2c-11eb-8c24-001a4aab830c@mon.smithi017' [6343689] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_adoption} ----------------------------------------------------------------- time: 00:08:42 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343689/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343689/ sentry: https://sentry.ceph.com/organizations/ceph/?query=093825a777064dc68bcd65fc8b061116 Command failed (workunit test cephadm/test_adoption.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [6343690] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:06:24 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343690/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343690/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67e7ff21c471425da53b35425494bfc4 Command failed on smithi120 with status 5: 'sudo systemctl stop ceph-5b25ee30-ff2d-11eb-8c24-001a4aab830c@mon.smithi120' [6343702] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:08:38 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343702/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343702/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b77682c857144d18188847089b9b029 Command failed (workunit test cephadm/test_cephadm.sh) on smithi161 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [6343709] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:08:12 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343709/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343709/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2065b929a06a44b6b901f702619c6885 Command failed on smithi002 with status 5: 'sudo systemctl stop ceph-16ff9bfe-ff30-11eb-8c24-001a4aab830c@mon.a' [6343711] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:31:52 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343711/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343711/ sentry: https://sentry.ceph.com/organizations/ceph/?query=573aa63b6db04ad7b9fe7af7f413c1f8 Command failed (workunit test rados/test.sh) on smithi019 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b5898830ff89e9ef894d372c8802cccce043e9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [6343715] rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_kubic_stable 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:05:27 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343715/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343715/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b4819c582d434fd7a904961c860fb57c Command failed on smithi033 with status 5: 'sudo systemctl stop ceph-f25f7bac-ff2f-11eb-8c24-001a4aab830c@mon.smithi033' [6343724] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_kubic_stable 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:06:21 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343724/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343724/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23af5545ddf7460798a0940bf7aa94de Command failed on smithi103 with status 5: 'sudo systemctl stop ceph-253a866a-ff31-11eb-8c24-001a4aab830c@mon.smithi103' [6343731] rados/cephadm/with-work/{0-distro/centos_8.3_kubic_stable fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:09:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343731/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343731/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ba18185cad746f99a9ed55553f3943f Command failed on smithi013 with status 5: 'sudo systemctl stop ceph-c494b910-ff31-11eb-8c24-001a4aab830c@mon.a' [6343733] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:08:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343733/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343733/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c4caf03bb264ae0859fd267ab340d5d Command failed on smithi174 with status 5: 'sudo systemctl stop ceph-50f21264-ff31-11eb-8c24-001a4aab830c@mon.a' [6343739] rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:08:29 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343739/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343739/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3951f9f7bd5b4016acb17dab02702982 Command failed on smithi083 with status 5: 'sudo systemctl stop ceph-e90f23de-ff31-11eb-8c24-001a4aab830c@mon.a' Dead ================================================================= [6343408] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:03:43 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343408/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f97c0486a5db4c2bb6a85f21826af780 {'Failure object was': {'smithi007.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4f f9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4f f9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} [6343435] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:02:29 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343435/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343435/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0fdc24113864facba152b008855086f {'Failure object was': {'smithi198.front.sepia.ceph.com': {'msg': 'non-zero return code', 'cmd': ['subscription-manager', 'repos', '-- enable=rhel-8-for-x86_64-baseos-rpms', '-- enable=rhel-8-for-x86_64-appstream-rpms'], 'stdout': '', 'stderr': 'HTTP error code 500: Internal error, contact administrator', 'rc': 70, 'start': '2021-08-17 05:20:01.231868', 'end': '2021-08-17 05:20:02.662801', 'delta': '0:00:01.430933', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms --enable=rhel-8-for-x86_64-appstream-rpms ', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': 'None', 'chdir': 'None', 'executable': 'None', 'creates': 'None', 'removes': 'None', 'stdin': 'None'}}, 'stdout_lines': [], 'stderr_lines': ['HTTP error code 500: Internal error, contact administrator'], '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4f f9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4f f9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'changed')"} [6343463] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343463/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343463/ hit max job timeout [6343512] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8.stream} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343512/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343512/ hit max job timeout [6343649] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343649/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343649/ hit max job timeout [6343734] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} ----------------------------------------------------------------- time: 00:03:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343734/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343734/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4390094c0d1346429cd2e677a913a8b1 {'Failure object was': {'smithi036.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4f f9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4f f9b63c7655015a3156de2c3b279/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} Running ================================================================= [6343746] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343746/ Pass ================================================================= [6343384] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_cls_all validater/lockdep} time: 00:16:24 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343384/ [6343385] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/scrub_test} time: 00:10:33 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343385/ [6343386] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:10 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343386/ [6343387] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 01:01:42 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343387/ [6343390] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/nfs-ingress 3-final} time: 00:22:39 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343390/ [6343391] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:14:47 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343391/ [6343392] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream}} time: 00:31:46 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343392/ [6343393] rados/dashboard/{centos_8.2_kubic_stable debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/dashboard} time: 01:10:33 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343393/ [6343394] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:17:37 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343394/ [6343395] rados/upgrade/parallel/{0-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:24:01 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343395/ [6343397] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:07:43 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343397/ [6343399] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:29:55 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343399/ [6343400] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:17:47 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343400/ [6343401] rados/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.3_kubic_stable} 2-node-mgr orchestrator_cli} time: 00:14:41 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343401/ [6343402] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:19:14 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343402/ [6343403] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8.stream}} time: 00:10:20 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343403/ [6343405] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:11:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343405/ [6343406] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:58:16 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343406/ [6343407] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:23:00 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343407/ [6343409] rados/cephadm/smoke-singlehost/{0-distro$/{rhel_8.3_kubic_stable} 1-start 2-services/basic 3-final} time: 00:17:16 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343409/ [6343410] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:35:49 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343410/ [6343411] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:27 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343411/ [6343412] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_8}} time: 00:13:23 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343412/ [6343413] rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.3-octopus 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/classic} time: 00:26:42 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343413/ [6343414] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} tasks/libcephsqlite} time: 00:17:09 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343414/ [6343415] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} tasks/mon_clock_with_skews} time: 00:08:26 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343415/ [6343416] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} time: 01:06:03 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343416/ [6343417] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:20:23 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343417/ [6343419] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:19:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343419/ [6343420] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:55:06 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343420/ [6343421] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_latest} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/radosbench} time: 00:56:00 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343421/ [6343422] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/nfs-ingress2 3-final} time: 00:19:13 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343422/ [6343423] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind} time: 00:43:40 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343423/ [6343425] rados/cephadm/with-work/{0-distro/rhel_8.3_kubic_stable fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:34:36 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343425/ [6343427] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:24 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343427/ [6343428] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects} time: 00:23:52 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343428/ [6343429] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream}} time: 00:09:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343429/ [6343431] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:09:10 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343431/ [6343432] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/nfs2 3-final} time: 00:19:49 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343432/ [6343433] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/progress} time: 00:31:09 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343433/ [6343436] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:25:53 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343436/ [6343437] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream}} time: 00:13:44 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343437/ [6343438] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:32:14 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343438/ [6343439] rados/cephadm/smoke/{distro/rhel_8.3_kubic_stable fixed-2 mon_election/connectivity start} time: 00:21:00 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343439/ [6343441] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/rgw-ingress 3-final} time: 00:20:54 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343441/ [6343442] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{rhel_8}} time: 00:33:28 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343442/ [6343443] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/one workloads/pool-create-delete} time: 00:29:23 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343443/ [6343444] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:48 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343444/ [6343445] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:32 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343445/ [6343446] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_cephadm_repos} time: 00:06:10 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343446/ [6343447] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:22:16 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343447/ [6343448] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:53 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343448/ [6343449] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} tasks/mon_recovery} time: 00:11:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343449/ [6343451] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:36:27 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343451/ [6343452] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/connectivity} time: 00:28:43 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343452/ [6343453] rados/standalone/{supported-random-distro$/{rhel_8} workloads/osd-backfill} time: 04:30:45 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343453/ [6343454] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:24:59 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343454/ [6343455] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/lockdep} time: 00:24:19 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343455/ [6343456] rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:37:21 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343456/ [6343457] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:09:39 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343457/ [6343458] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:22:13 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343458/ [6343459] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:07:22 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343459/ [6343460] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_8.stream}} time: 00:09:40 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343460/ [6343461] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8.stream} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:23:20 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343461/ [6343464] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/basic 3-final} time: 00:19:56 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343464/ [6343466] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:10:18 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343466/ [6343467] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/rados_cls_all} time: 00:14:03 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343467/ [6343468] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:47 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343468/ [6343469] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8.stream} tasks/prometheus} time: 00:13:08 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343469/ [6343470] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_latest} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/rbd_cls} time: 00:20:36 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343470/ [6343472] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:34:20 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343472/ [6343473] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:23:34 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343473/ [6343474] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:16:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343474/ [6343475] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:20:56 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343475/ [6343476] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:14:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343476/ [6343477] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/client-keyring 3-final} time: 00:16:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343477/ [6343478] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:18 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343478/ [6343479] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:41:43 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343479/ [6343480] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/sync-many workloads/rados_5925} time: 00:11:27 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343480/ [6343481] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 1-rook 2-workload/none 3-final cluster/1-node k8s/1.21 net/calico rook/master} time: 00:11:01 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343481/ [6343483] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:37 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343483/ [6343484] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:05 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343484/ [6343485] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:10:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343485/ [6343488] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:31 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343488/ [6343489] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:27:31 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343489/ [6343490] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream}} time: 00:10:59 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343490/ [6343492] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:19:14 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343492/ [6343493] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/mirror 3-final} time: 00:19:58 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343493/ [6343495] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:08 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343495/ [6343497] rados/standalone/{supported-random-distro$/{centos_8.stream} workloads/osd} time: 03:12:38 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343497/ [6343498] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} tasks/rados_python} time: 00:15:08 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343498/ [6343499] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:29:01 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343499/ [6343500] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream}} time: 00:41:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343500/ [6343501] rados/cephadm/smoke-singlehost/{0-distro$/{rhel_8.3_kubic_stable} 1-start 2-services/rgw 3-final} time: 00:17:46 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343501/ [6343502] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_8.stream}} time: 00:12:49 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343502/ [6343503] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/classic} time: 00:28:48 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343503/ [6343504] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8.stream} tasks/workunits} time: 00:10:36 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343504/ [6343506] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:10:17 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343506/ [6343507] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:35:39 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343507/ [6343508] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:12:13 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343508/ [6343509] rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:25:20 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343509/ [6343510] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:35:49 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343510/ [6343513] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:14:39 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343513/ [6343514] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/nfs-ingress-rgw 3-final} time: 00:18:17 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343514/ [6343516] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_api_tests} time: 00:25:50 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343516/ [6343517] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:35:35 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343517/ [6343518] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343518/ [6343519] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:28 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343519/ [6343521] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:34:50 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343521/ [6343522] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:08:11 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343522/ [6343523] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_latest} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:34:29 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343523/ [6343524] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:24:42 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343524/ [6343526] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:15:04 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343526/ [6343528] rados/singleton/{all/peer mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream}} time: 00:09:24 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343528/ [6343529] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:11:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343529/ [6343530] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:21:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343530/ [6343531] rados/cephadm/smoke/{distro/rhel_8.3_kubic_stable fixed-2 mon_election/classic start} time: 00:19:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343531/ [6343532] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:32:55 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343532/ [6343533] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:10:53 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343533/ [6343534] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:29 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343534/ [6343535] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/nfs-ingress2 3-final} time: 00:22:41 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343535/ [6343536] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} time: 02:36:33 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343536/ [6343537] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_cephadm_repos} time: 00:06:24 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343537/ [6343538] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:10:05 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343538/ [6343539] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/crash} time: 00:11:47 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343539/ [6343540] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/nfs 3-final} time: 00:16:42 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343540/ [6343541] rados/objectstore/{backends/objectstore-filestore-memstore supported-random-distro$/{centos_8.stream}} time: 00:29:26 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343541/ [6343542] rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.3-octopus 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/connectivity} time: 00:27:39 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343542/ [6343543] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:18:48 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343543/ [6343544] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:29 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343544/ [6343545] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:28:49 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343545/ [6343547] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:10:08 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343547/ [6343549] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/rados_striper} time: 00:12:06 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343549/ [6343550] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} time: 00:41:37 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343550/ [6343552] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8.stream}} time: 00:29:45 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343552/ [6343553] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 01:01:36 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343553/ [6343554] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:35 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343554/ [6343555] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:32:38 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343555/ [6343557] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:08:10 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343557/ [6343558] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:10:03 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343558/ [6343559] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:18:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343559/ [6343560] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:16:43 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343560/ [6343561] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:31:20 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343561/ [6343562] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:21:55 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343562/ [6343563] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados tasks/rados_api_tests validater/valgrind} time: 01:08:42 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343563/ [6343564] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/rgw-ingress 3-final} time: 00:23:38 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343564/ [6343565] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:19 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343565/ [6343566] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache} time: 00:12:50 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343566/ [6343568] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:28:21 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343568/ [6343569] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:23:15 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343569/ [6343570] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/rgw 3-final} time: 00:17:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343570/ [6343571] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 1-rook 2-workload/none 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.0} time: 00:11:02 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343571/ [6343572] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/failover} time: 00:14:40 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343572/ [6343573] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_latest} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:21:12 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343573/ [6343575] rados/dashboard/{centos_8.2_kubic_stable debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/dashboard} time: 01:10:41 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343575/ [6343577] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:12:05 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343577/ [6343578] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:26:45 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343578/ [6343579] rados/standalone/{supported-random-distro$/{centos_8} workloads/c2c} time: 00:13:51 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343579/ [6343581] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:24:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343581/ [6343583] rados/singleton/{all/random-eio mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream}} time: 00:22:15 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343583/ [6343584] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/dedup-io-mixed} time: 00:13:01 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343584/ [6343585] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} tasks/rados_workunit_loadgen_big} time: 00:30:59 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343585/ [6343587] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:39:06 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343587/ [6343588] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:08:48 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343588/ [6343589] rados/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.3_kubic_stable} 2-node-mgr orchestrator_cli} time: 00:25:28 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343589/ [6343590] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_8.stream}} time: 00:10:26 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343590/ [6343591] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:14:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343591/ [6343593] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:25:37 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343593/ [6343594] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} time: 00:12:56 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343594/ [6343595] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:21:33 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343595/ [6343596] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343596/ [6343598] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:18:05 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343598/ [6343599] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/classic} time: 00:29:48 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343599/ [6343600] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343600/ [6343601] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream}} time: 00:23:48 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343601/ [6343602] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:23 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343602/ [6343604] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/lockdep} time: 00:24:50 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343604/ [6343605] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:39:17 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343605/ [6343606] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/client-keyring 3-final} time: 00:19:59 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343606/ [6343607] rados/cephadm/with-work/{0-distro/rhel_8.3_kubic_stable fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python} time: 00:35:09 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343607/ [6343608] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mix} time: 00:24:05 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343608/ [6343609] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:12:04 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343609/ [6343610] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:07:01 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343610/ [6343611] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:21:38 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343611/ [6343612] rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:07:21 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343612/ [6343613] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8.stream}} time: 00:14:02 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343613/ [6343614] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/iscsi 3-final} time: 00:16:35 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343614/ [6343615] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:24:26 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343615/ [6343617] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:24:17 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343617/ [6343619] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:18:23 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343619/ [6343620] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-radosbench} time: 01:58:48 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343620/ [6343621] rados/singleton/{all/test-crash mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:10:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343621/ [6343622] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:17:56 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343622/ [6343624] rados/cephadm/smoke/{distro/rhel_8.3_kubic_stable fixed-2 mon_election/connectivity start} time: 00:20:24 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343624/ [6343625] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/one workloads/snaps-few-objects} time: 00:35:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343625/ [6343626] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:26:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343626/ [6343627] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_latest} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:33:45 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343627/ [6343628] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/classic task/test_cephadm_repos} time: 00:06:13 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343628/ [6343629] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:14 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343629/ [6343630] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:21:42 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343630/ [6343631] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:28:15 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343631/ [6343632] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:18 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343632/ [6343633] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:06:16 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343633/ [6343634] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/nfs-ingress-rgw 3-final} time: 00:21:48 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343634/ [6343635] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:11:19 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343635/ [6343636] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/connectivity} time: 00:28:01 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343636/ [6343637] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:20 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343637/ [6343638] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:29:04 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343638/ [6343640] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/nfs-ingress 3-final} time: 00:18:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343640/ [6343641] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/radosbench} time: 01:05:19 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343641/ [6343642] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream}} time: 01:25:15 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343642/ [6343643] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:58 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343643/ [6343644] rados/cephadm/with-work/{0-distro/rhel_8.3_kubic_stable fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:48:05 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343644/ [6343645] rados/objectstore/{backends/filejournal supported-random-distro$/{ubuntu_latest}} time: 00:06:36 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343645/ [6343646] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:27:16 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343646/ [6343650] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect} time: 00:26:18 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343650/ [6343651] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:03 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343651/ [6343653] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:16:59 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343653/ [6343654] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} time: 00:20:10 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343654/ [6343655] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:41:33 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343655/ [6343656] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 01:26:31 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343656/ [6343658] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:09:19 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343658/ [6343659] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:30:02 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343659/ [6343660] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:14:45 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343660/ [6343662] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:59 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343662/ [6343663] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/sync-many workloads/pool-create-delete} time: 00:18:06 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343663/ [6343664] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} tasks/readwrite} time: 00:12:28 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343664/ [6343666] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:18 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343666/ [6343667] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:01 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343667/ [6343668] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:50 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343668/ [6343669] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/nfs 3-final} time: 00:19:55 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343669/ [6343670] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} tasks/mon_recovery} time: 00:20:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343670/ [6343672] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/nfs2 3-final} time: 00:15:53 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343672/ [6343673] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:24:32 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343673/ [6343674] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:36 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343674/ [6343675] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/lockdep} time: 00:11:58 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343675/ [6343677] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8.stream}} time: 00:10:44 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343677/ [6343678] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:19:02 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343678/ [6343679] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8.stream}} time: 02:13:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343679/ [6343681] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:18:53 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343681/ [6343682] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_latest} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:20:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343682/ [6343684] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:17:54 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343684/ [6343685] rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.3-octopus 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/classic} time: 00:27:32 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343685/ [6343686] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343686/ [6343687] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:11:12 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343687/ [6343688] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:13:53 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343688/ [6343691] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:06:19 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343691/ [6343692] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} tasks/repair_test} time: 00:16:50 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343692/ [6343693] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read} time: 00:21:46 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343693/ [6343694] rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:25:44 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343694/ [6343695] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:23:25 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343695/ [6343696] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} time: 00:19:19 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343696/ [6343697] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/rgw 3-final} time: 00:21:21 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343697/ [6343698] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:12:57 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343698/ [6343699] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:25:18 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343699/ [6343700] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_5925} time: 00:11:42 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343700/ [6343701] rados/singleton/{all/deduptool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream}} time: 00:09:33 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343701/ [6343703] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:39:01 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343703/ [6343704] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:48 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343704/ [6343705] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:18:11 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343705/ [6343706] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/basic 3-final} time: 00:15:59 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343706/ [6343707] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:46 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343707/ [6343708] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:21:53 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343708/ [6343710] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:21:37 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343710/ [6343712] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8}} time: 02:22:15 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343712/ [6343713] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:12:24 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343713/ [6343714] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:07:30 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343714/ [6343716] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/prometheus} time: 00:22:35 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343716/ [6343717] rados/cephadm/smoke/{distro/rhel_8.3_kubic_stable fixed-2 mon_election/classic start} time: 00:21:29 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343717/ [6343718] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:27:45 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343718/ [6343719] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:27:46 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343719/ [6343720] rados/cephadm/workunits/{0-distro/centos_8.2_kubic_stable mon_election/connectivity task/test_cephadm_repos} time: 00:06:09 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343720/ [6343721] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/scrub_test} time: 00:14:07 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343721/ [6343722] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects} time: 00:20:21 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343722/ [6343723] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:09:49 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343723/ [6343725] rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/connectivity} time: 00:28:46 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343725/ [6343726] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:17 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343726/ [6343727] rados/cephadm/smoke-roleless/{0-distro/rhel_8.3_kubic_stable 1-start 2-services/iscsi 3-final} time: 00:20:38 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343727/ [6343728] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:21:16 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343728/ [6343729] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_latest} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:18:27 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343729/ [6343730] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:10:05 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343730/ [6343732] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} time: 00:25:11 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343732/ [6343735] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:25:00 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343735/ [6343736] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:21 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343736/ [6343737] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:08 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343737/ [6343738] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:28:33 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343738/ [6343740] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:53:12 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343740/ [6343741] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:18:16 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343741/ [6343742] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream}} time: 00:28:22 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343742/ [6343743] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:49 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343743/ [6343744] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 1-start 2-services/mirror 3-final} time: 00:17:10 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343744/ [6343745] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:05:41 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343745/ [6343747] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream}} time: 00:31:51 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343747/ [6343748] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/lockdep} time: 00:16:22 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343748/ [6343749] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:16:34 info: http://pulpito.ceph.com/kchai-2021-08-17_04:49:07-rados-wip-kefu-testing-2021-08-17-0902-distro-basic-smithi/6343749/