2021-07-24T16:13:22.302 INFO:root:teuthology version: 1.1.0-b6f3efab 2021-07-24T16:13:22.303 INFO:teuthology.results:subset: '10130/99999' 2021-07-24T16:13:22.304 INFO:teuthology.results:seed: '2477' 2021-07-24T17:02:30.651 INFO:root:teuthology version: 1.1.0-b6f3efab 2021-07-24T17:02:30.685 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-07-25T02:45:13.592 INFO:teuthology.results:Tests finished! gathering results... 2021-07-25T02:45:20.773 INFO:scrape:Found 74 jobs 2021-07-25T02:45:22.786 WARNING:scrape:Not-understood crash indication 2021-07-24T16:53:33.027 INFO:tasks.rgw.c2.client.0.smithi170.stderr:daemon-helper: command crashed with signal 11 2021-07-25T02:45:22.787 WARNING:scrape:Not-understood crash indication 2021-07-25T02:45:23.947 WARNING:scrape:Not-understood crash indication 2021-07-24T16:55:26.646 INFO:tasks.rgw.c2.client.0.smithi104.stderr:daemon-helper: command crashed with signal 11 2021-07-25T02:45:23.948 WARNING:scrape:Not-understood crash indication 2021-07-25T02:45:25.967 WARNING:scrape:Not-understood crash indication 2021-07-24T16:55:27.498 INFO:tasks.rgw.c2.client.0.smithi114.stderr:daemon-helper: command crashed with signal 11 2021-07-25T02:45:25.968 WARNING:scrape:Not-understood crash indication 2021-07-25T02:45:26.350 WARNING:scrape:Not-understood crash indication 2021-07-24T17:26:43.663 INFO:tasks.rgw.c2.client.0.smithi163.stderr:daemon-helper: command crashed with signal 11 2021-07-25T02:45:26.351 WARNING:scrape:Not-understood crash indication 2021-07-25T02:45:28.692 INFO:scrape:Found 6 distinct failure reasons 2021-07-25T02:45:28.693 INFO:scrape:Failure: rgw multisite test failures 2021-07-25T02:45:28.693 INFO:scrape:7 jobs: ['6290199', '6290214', '6290177', '6290179', '6290215', '6290197', '6290234'] 2021-07-25T02:45:28.694 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks'] 2021-07-25T02:45:28.694 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'supported-random-distro$/{ubuntu_latest}', 'tasks', 'upgrade/octopus-x/rgw-multisite/{clusters', 'upgrade/pacific-x/rgw-multisite/{clusters', 'upgrade/primary}', 'upgrade/secondary}'] 2021-07-25T02:45:28.695 INFO:scrape: 2021-07-25T02:45:28.695 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rgw.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' 2021-07-25T02:45:28.696 INFO:scrape:8 jobs: ['6290195', '6290232', '6290176', '6290223', '6290203', '6290213', '6290184', '6290166'] 2021-07-25T02:45:28.696 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-07-25T02:45:28.696 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade/octopus-x/parallel/{0-distro/centos_8.3_kubic_stable', 'upgrade/octopus-x/parallel/{0-distro/rhel_8.3_kubic_stable', 'workload/{ec-rados-default'] 2021-07-25T02:45:28.697 INFO:scrape: 2021-07-25T02:45:28.697 INFO:scrape:Failure: Command failed on smithi084 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' 2021-07-25T02:45:28.698 INFO:scrape:21 jobs: ['6290229', '6290200', '6290227', '6290206', '6290219', '6290169', '6290171', '6290222', '6290209', '6290192', '6290208', '6290226', '6290165', '6290220', '6290189', '6290172', '6290182', '6290188', '6290202', '6290186', '6290183'] 2021-07-25T02:45:28.698 INFO:scrape:suites intersection: ['rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2021-07-25T02:45:28.699 INFO:scrape:suites union: ['0-roles', '1-ceph-install/octopus', '1-start', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '2-partial-upgrade/firsthalf', '3-stress-tasks/{radosbench', '3-thrash/default', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack', 'upgrade/pacific-x/stress-split/{0-distro/centos_8.3_kubic_stable', 'upgrade/pacific-x/stress-split/{0-distro/rhel_8.3_kubic_stable', 'upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04'] 2021-07-25T02:45:28.699 INFO:scrape: 2021-07-25T02:45:28.700 INFO:scrape:Dead: b'2021-07-25T02:42:43.802 DEBUG:teuthology.exit:Finished running handlers' 2021-07-25T02:45:28.700 INFO:scrape:12 jobs: ['6290221', '6290224', '6290230', '6290212', '6290174', '6290167', '6290211', '6290204', '6290185', '6290194', '6290190', '6290175'] 2021-07-25T02:45:28.701 INFO:scrape:suites intersection: [] 2021-07-25T02:45:28.701 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/octopus-x/stress-split/{0-distro/centos_8.3_kubic_stable', 'upgrade/octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable', 'upgrade/pacific-x/parallel/{0-distro$/{rhel_8.3_kubic_stable}', 'workload/{ec-rados-default'] 2021-07-25T02:45:28.702 INFO:scrape: 2021-07-25T02:45:28.702 INFO:scrape:Failure: Command failed on smithi151 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd' 2021-07-25T02:45:28.703 INFO:scrape:1 jobs: ['6290162'] 2021-07-25T02:45:28.703 INFO:scrape:suites: ['frontend', 'overrides', 'realm', 'supported-random-distro$/{centos_8}', 'tasks', 'upgrade/pacific-x/rgw-multisite/{clusters', 'upgrade/primary}'] 2021-07-25T02:45:28.703 INFO:scrape: 2021-07-25T02:45:28.704 INFO:scrape:Failure: {'Failure object was': {'smithi057.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} 2021-07-25T02:45:28.704 INFO:scrape:1 jobs: ['6290163'] 2021-07-25T02:45:28.705 INFO:scrape:suites: ['1-ceph-install/octopus', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'mon_election/connectivity', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2021-07-25T02:45:28.705 INFO:scrape: 2021-07-25T02:45:28.711 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/ failed: 37 dead: 13 running: 0 waiting: 0 queued: 0 passed: 24 Fail ================================================================= [6290162] upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm supported-random-distro$/{centos_8} tasks upgrade/primary} ----------------------------------------------------------------- time: 00:11:35 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290162/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290162/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba5d8bdcfbb14831a7b02c8c1e3c3490 Command failed on smithi151 with status 1: 'sudo yum -y upgrade ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd- mirror rbd-nbd' [6290165] upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:15:10 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290165/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290165/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c85575138ad4ef3a26bccc4a8263a29 Command failed on smithi098 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290166] upgrade/octopus-x/parallel/{0-distro/rhel_8.3_kubic_stable 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:46:23 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290166/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290166/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f4cba48ba59468cb97327fd78fe559a Command failed (workunit test cls/test_cls_rgw.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6290169] upgrade/pacific-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:17:17 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290169/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290169/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2b35133d828467d9dd44479f84493b3 Command failed on smithi075 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290171] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:58:56 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290171/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290171/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c6eb5077a604befaf19b376541b8793 Command failed on smithi191 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6290172] upgrade/pacific-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:26:28 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290172/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290172/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8fe9c7e25f1947dfbb46ca641b345f0e Command failed on smithi043 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290176] upgrade/octopus-x/parallel/{0-distro/centos_8.3_kubic_stable 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:37:06 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290176/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a990f8f2816e416a8f87b56f982a6492 Command failed (workunit test cls/test_cls_rgw.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6290177] upgrade/octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:10:28 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290177/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290177/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb6cb122125a44a5bd20d3aacd41f1f1 rgw multisite test failures [6290179] upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm supported-random-distro$/{ubuntu_latest} tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:00:25 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290179/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290179/ sentry: https://sentry.ceph.com/organizations/ceph/?query=416cab7d0ef04006b3eed2ddbad9cd4a rgw multisite test failures [6290182] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 02:09:34 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290182/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290182/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57ccc83300584795be5f2f34e47615f6 Command failed on smithi153 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6290183] upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:16:29 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290183/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290183/ sentry: https://sentry.ceph.com/organizations/ceph/?query=654b2467878f46a8b52ea07a147b0c54 Command failed on smithi169 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290184] upgrade/octopus-x/parallel/{0-distro/rhel_8.3_kubic_stable 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:46:06 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290184/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290184/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b56d695766564bd196bdae693619a82b Command failed (workunit test cls/test_cls_rgw.sh) on smithi106 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6290186] upgrade/pacific-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:16:12 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290186/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de8a696b167f4aebae7230f3997e90e3 Command failed on smithi038 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290188] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 04:53:14 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290188/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290188/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b182e1b65094b9b94dda34bafd4e23b Command failed on smithi196 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6290189] upgrade/pacific-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:26:59 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290189/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa74fe515be441f0ab3534752a6f8bfa Command failed on smithi035 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290192] upgrade/pacific-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:26:44 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290192/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290192/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db81bb02204a45db810b9b529ddfce61 Command failed on smithi027 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290195] upgrade/octopus-x/parallel/{0-distro/centos_8.3_kubic_stable 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:43 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290195/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d3f061bc1d046b891ee5e85a1229313 Command failed (workunit test cls/test_cls_rgw.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6290197] upgrade/octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 03:09:33 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290197/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290197/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df2cee5047504cc9b390bb2836e923d7 rgw multisite test failures [6290199] upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm supported-random-distro$/{ubuntu_latest} tasks upgrade/primary} ----------------------------------------------------------------- time: 03:06:09 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290199/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290199/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4c3554f99d9428cbd89407bfafc812f rgw multisite test failures [6290200] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:52:57 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290200/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290200/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45c53b2ebdc0404382eddba290ff07d6 Command failed on smithi197 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6290202] upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:15:59 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290202/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd2579eca08a48a8a42f26c8a739cab4 Command failed on smithi008 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290203] upgrade/octopus-x/parallel/{0-distro/rhel_8.3_kubic_stable 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:45:43 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290203/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290203/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b10c46c5ddd147e58675522ff46d611c Command failed (workunit test cls/test_cls_rgw.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6290206] upgrade/pacific-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:16:16 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290206/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290206/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4432f3fb82db44789abcc213ba7ad9fb Command failed on smithi112 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290208] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:56:34 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290208/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290208/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6792d36cc64348cbbad7d4fdb89accdb Command failed on smithi136 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6290209] upgrade/pacific-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:26:31 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290209/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c7f845393dca44869067dbdb2f562696 Command failed on smithi002 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290213] upgrade/octopus-x/parallel/{0-distro/centos_8.3_kubic_stable 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:37:37 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290213/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290213/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb74bd11c9cb4930a54a690f2f4bb317 Command failed (workunit test cls/test_cls_rgw.sh) on smithi012 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6290214] upgrade/octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:07:53 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290214/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290214/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c3558ccd6404f9c9a35d0400679a6de rgw multisite test failures [6290215] upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm supported-random-distro$/{ubuntu_latest} tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:00:48 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290215/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290215/ sentry: https://sentry.ceph.com/organizations/ceph/?query=455fc9dbb9e0404dae1b14dfea9d2d13 rgw multisite test failures [6290219] upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:14:47 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290219/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290219/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e9f40795946470ea503b5c9f5f871be Command failed on smithi037 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290220] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:36:34 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290220/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290220/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b685549c7cdc44199aed22a3d712d127 Command failed on smithi194 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6290222] upgrade/pacific-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:15:01 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290222/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290222/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54e17fdcd3de4f22ad1d1e6588d4ebbc Command failed on smithi112 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290223] upgrade/octopus-x/parallel/{0-distro/rhel_8.3_kubic_stable 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:45:05 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290223/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290223/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba7c8d6b98e5475fb4b70135470d82e0 Command failed (workunit test cls/test_cls_rgw.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6290226] upgrade/pacific-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:15:32 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290226/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290226/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2bc6b9eb0c5f4929ad9f164065fbda0e Command failed on smithi169 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290227] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:52:10 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290227/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290227/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02bfa359227842e2abe9f5c76cedefaf Command failed on smithi180 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6290229] upgrade/pacific-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:24:57 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290229/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290229/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f095d399ad340dbbb04094d7286c33d Command failed on smithi084 with status 13: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json' [6290232] upgrade/octopus-x/parallel/{0-distro/centos_8.3_kubic_stable 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:09 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290232/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290232/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c236c7ab638a45979764ea84b819e431 Command failed (workunit test cls/test_cls_rgw.sh) on smithi035 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' [6290234] upgrade/octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 03:08:31 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290234/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290234/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07945fc330514e7c8a6ed95e46a0197e rgw multisite test failures Dead ================================================================= [6290163] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 00:06:12 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290163/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290163/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98f5884d82424943a06f429df6da483a {'Failure object was': {'smithi057.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70 e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70 e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} [6290167] upgrade/octopus-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290167/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290167/ [6290174] upgrade/octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290174/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290174/ [6290175] upgrade/pacific-x/parallel/{0-distro$/{rhel_8.3_kubic_stable} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290175/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290175/ [6290185] upgrade/octopus-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290185/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290185/ [6290190] upgrade/octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290190/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290190/ [6290194] upgrade/octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290194/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290194/ [6290204] upgrade/octopus-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290204/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290204/ [6290211] upgrade/octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290211/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290211/ [6290212] upgrade/pacific-x/parallel/{0-distro$/{rhel_8.3_kubic_stable} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290212/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290212/ [6290221] upgrade/octopus-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290221/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290221/ [6290224] upgrade/octopus-x/stress-split/{0-distro/centos_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290224/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290224/ [6290230] upgrade/octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290230/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290230/ Pass ================================================================= [6290164] upgrade/octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:44:19 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290164/ [6290168] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:36 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290168/ [6290170] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:45:07 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290170/ [6290173] upgrade/octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:08 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290173/ [6290178] upgrade/octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:00:29 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290178/ [6290180] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:42:51 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290180/ [6290181] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:20:23 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290181/ [6290187] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 04:07:57 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290187/ [6290191] upgrade/octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:46 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290191/ [6290193] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:10 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290193/ [6290196] upgrade/pacific-x/parallel/{0-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:27:41 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290196/ [6290198] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 01:44:47 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290198/ [6290201] upgrade/octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:56:20 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290201/ [6290205] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:14 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290205/ [6290207] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:14:43 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290207/ [6290210] upgrade/octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:17 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290210/ [6290216] upgrade/octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:13:48 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290216/ [6290217] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:06:10 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290217/ [6290218] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:18:43 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290218/ [6290225] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 03:34:37 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290225/ [6290228] upgrade/octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:12 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290228/ [6290231] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:25:26 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290231/ [6290233] upgrade/pacific-x/parallel/{0-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:25:16 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290233/ [6290235] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:09:23 info: http://pulpito.ceph.com/kchai-2021-07-24_16:12:52-upgrade-wip-kefu-testing-2021-07-24-2153-distro-basic-smithi/6290235/