2021-12-26T15:16:25.867 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-26T15:16:25.869 INFO:teuthology.results:subset: '100/100000'
2021-12-26T15:16:25.870 INFO:teuthology.results:seed: '8691'
2021-12-26T16:56:00.671 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-26T16:56:00.713 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-12-27T04:49:57.313 INFO:teuthology.results:Tests finished! gathering results...
2021-12-27T04:50:14.128 INFO:scrape:Found 170 jobs
2021-12-27T04:50:22.481 INFO:scrape:Found 4 distinct failure reasons
2021-12-27T04:50:22.482 INFO:scrape:Failure: {'Failure object was': {'smithi112.front.sepia.ceph.com': {'results': [{'msg': 'non-zero return code', 'cmd': ['dnf', '-y', 'copr', 'enable', 'ceph/python3-asyncssh'], 'stdout': 'Updating Subscription Management repositories.', 'stderr': "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nCurl error (7): Couldn't connect to server for https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 [Failed to connect to copr.fedorainfracloud.org port 443: Connection timed out]", 'rc': 1, 'start': '2021-12-26 15:29:34.269811', 'end': '2021-12-26 15:30:34.929213', 'delta': '0:01:00.659402', 'changed': True, 'failed': True, 'invocation': {'module_args': {'_raw_params': 'dnf -y copr enable ceph/python3-asyncssh', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': 'None', 'chdir': 'None', 'executable': 'None', 'creates': 'None', 'removes': 'None', 'stdin': 'None'}}, 'stdout_lines': ['Updating Subscription Management repositories.'], 'stderr_lines': ['Enabling a Copr repository. Please note that this repository is not part', 'of the main distribution, and quality may vary.', '', 'The Fedora Project does not exercise any power over the contents of', 'this repository beyond the rules outlined in the Copr FAQ at', '<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,', 'and packages are not held to any quality or security level.', '', 'Please do not file bug reports about these packages in Fedora', 'Bugzilla. In case of problems, contact the owner of this repository.', "Curl error (7): Couldn't connect to server for https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 [Failed to connect to copr.fedorainfracloud.org port 443: Connection timed out]"], '_ansible_no_log': False, 'item': 'ceph/python3-asyncssh', 'ansible_loop_var': 'item', '_ansible_item_label': 'ceph/python3-asyncssh'}], 'changed': True, 'msg': 'All items completed', 'warnings': ["Consider using the dnf module rather than running 'dnf'.  If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 199, in represent_list return self.represent_sequence(\'tag:yaml.org,2002:seq\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 92, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'ceph/python3-asyncssh')"}
2021-12-27T04:50:22.483 INFO:scrape:37 jobs
2021-12-27T04:50:22.484 INFO:scrape:suites intersection: []
2021-12-27T04:50:22.484 INFO:scrape:suites union: ['0-nvme-loop', '0-size-min-size-overrides/2-size-2-min-size', '1-install/nautilus-v2only', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '1-ranks/1', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-node-mgr', '2-ops/repave-all}', '2-ops/rmdir-reactivate}', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-services/nfs', '2-services/nfs2', '2-services/rgw', '2-thrash', '3-final}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-tasks/rados_api_tests', '3-upgrade-with-workload', '3-verify}', '4-verify}}', 'agent/on', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'centos_8.stream_container_tools', 'ceph', 'cluster/{12-osds', 'clusters/9-mons', 'clusters/{fixed-2', 'clusters/{fixed-4', 'clusters/{openstack', 'conf/{client', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'debug/mgr', 'fast/fast', 'fast/normal', 'fixed-2', 'mds', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'orchestrator_cli}', 'osd}', 'overrides/{pg-warn', 'rados', 'rados/basic/{ceph', 'rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'rados/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_3.0}', 'rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.2_container_tools_3.0', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.2_container_tools_3.0}', 'rados/cephadm/smoke/{0-distro/centos_8.3_container_tools_3.0', 'rados/cephadm/thrash/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/with-work/{0-distro/centos_8.2_container_tools_3.0', 'rados/cephadm/with-work/{0-distro/centos_8.3_container_tools_3.0', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/monthrash/{ceph', 'rados/objectstore/{backends/filejournal', 'rados/objectstore/{backends/filestore-idempotent-aio-journal', 'rados/singleton-nomsgr/{all/health-warnings', 'rados/singleton-nomsgr/{all/large-omap-object-warnings', 'rados/singleton-nomsgr/{all/librados_hello_world', 'rados/singleton/{all/mon-auth-caps', 'rados/singleton/{all/mon-config-keys', 'rados/singleton/{all/osd-backfill', 'rados/singleton/{all/osd-recovery-incomplete', 'rados/standalone/{supported-random-distro$/{centos_8}', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash-old-clients/{0-distro$/{centos_8.3_container_tools_3.0}', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'roles', 'root}', 'start', 'start}', 'supported-random-distro$/{centos_8.stream}}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'syntax', 'tasks/module_selftest}', 'tasks/mon_recovery', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rados_workunit_loadgen_big}', 'tasks/{0-from/v16.2.4', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/one', 'thrashers/pggrow', 'thrashosds-health', 'thrashosds-health}', 'three-plus-one}', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-fast-read}', 'workloads/erasure-code}', 'workloads/rados_mon_workunits}', 'workloads/redirect_set_object}', 'workloads/set-chunks-read}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/test_rbd_api}']
2021-12-27T04:50:22.485 INFO:scrape:
2021-12-27T04:50:22.485 INFO:scrape:Failure: Command failed (workunit test osd-backfill/osd-backfill-stats.sh) on smithi086 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0ddea0a93bb5fe35d853b01c4d53dd48ccb49b1e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh'
2021-12-27T04:50:22.486 INFO:scrape:1 jobs: ['6585706']
2021-12-27T04:50:22.486 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_8}', 'workloads/osd-backfill}']
2021-12-27T04:50:22.487 INFO:scrape:
2021-12-27T04:50:22.487 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
2021-12-27T04:50:22.488 INFO:scrape:1 jobs: ['6585655']
2021-12-27T04:50:22.488 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}']
2021-12-27T04:50:22.489 INFO:scrape:
2021-12-27T04:50:22.489 INFO:scrape:Dead: b'2021-12-27T03:39:30.923 DEBUG:teuthology.exit:Finished running handlers'
2021-12-27T04:50:22.490 INFO:scrape:2 jobs: ['6585648', '6585748']
2021-12-27T04:50:22.490 INFO:scrape:suites intersection: ['1-bootstrap/octopus', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools']
2021-12-27T04:50:22.491 INFO:scrape:suites union: ['1-bootstrap/octopus', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools']
2021-12-27T04:50:22.491 INFO:scrape: