2021-03-02T23:11:37.434 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-02T23:11:37.436 INFO:teuthology.results:seed: '1727' 2021-03-02T23:13:14.658 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-02T23:13:14.660 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-02T23:59:18.137 INFO:teuthology.results:Tests finished! gathering results... 2021-03-02T23:59:23.062 INFO:scrape:Found 9 jobs 2021-03-02T23:59:24.096 WARNING:scrape:Missing log /home/teuthworker/archive/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928396/remote/smithi172/log/ceph-mds.f.log.gz 2021-03-02T23:59:24.097 WARNING:scrape:Not-understood crash indication 2021-03-02T23:59:24.114 WARNING:scrape:Missing log /home/teuthworker/archive/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928396/remote/smithi172/log/ceph-mds.f.log.gz 2021-03-02T23:59:24.115 WARNING:scrape:Not-understood crash indication 2021-03-02T23:59:24.131 WARNING:scrape:Missing log /home/teuthworker/archive/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928396/remote/smithi172/log/ceph-mds.f.log.gz 2021-03-02T23:59:24.132 WARNING:scrape:Not-understood crash indication 2021-03-02T23:59:24.138 WARNING:scrape:Misunderstood line: 2021-03-02T23:59:24.199 WARNING:scrape:Missing log /home/teuthworker/archive/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928403/remote/smithi174/log/ceph-mds.b.log.gz 2021-03-02T23:59:24.200 WARNING:scrape:Missing log /home/teuthworker/archive/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928403/remote/smithi174/log/ceph-mds.d.log.gz 2021-03-02T23:59:24.200 WARNING:scrape:Not-understood crash indication 2021-03-02T23:59:24.206 WARNING:scrape:Misunderstood line: 2021-03-02T23:59:24.249 WARNING:scrape:Missing log /home/teuthworker/archive/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928399/remote/smithi045/log/ceph-mds.e.log.gz 2021-03-02T23:59:24.250 WARNING:scrape:Not-understood crash indication 2021-03-02T23:59:24.255 WARNING:scrape:Misunderstood line: 2021-03-02T23:59:24.408 INFO:scrape:Found 6 distinct failure reasons 2021-03-02T23:59:24.409 INFO:scrape:Failure: object of type 'generator' has no len() 2021-03-02T23:59:24.409 INFO:scrape:1 jobs: ['5928402'] 2021-03-02T23:59:24.409 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'overrides}}', 'ranks/5', 'session_timeout', 'tasks/dbench', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-02T23:59:24.410 INFO:scrape: 2021-03-02T23:59:24.410 INFO:scrape:Valgrind: mds (InvalidJump) 2021-03-02T23:59:24.411 INFO:scrape:3 jobs: ['5928396', '5928403', '5928399'] 2021-03-02T23:59:24.411 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{frag_enable', 'overrides}}', 'session_timeout', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-02T23:59:24.412 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'overrides}}', 'ranks/1', 'ranks/3', 'ranks/5', 'session_timeout', 'tasks/dbench', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-02T23:59:24.412 INFO:scrape: 2021-03-02T23:59:24.413 INFO:scrape:Failure: {'smithi203.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi203.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi203.front.sepia.ceph.com port 22: No route to host\r\n', 'unreachable': True}} 2021-03-02T23:59:24.413 INFO:scrape:1 jobs: ['5928404'] 2021-03-02T23:59:24.413 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag_enable', 'overrides}}', 'ranks/3', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-02T23:59:24.414 INFO:scrape: 2021-03-02T23:59:24.414 INFO:scrape:Failure: Command failed on smithi039 with status 100: 'sudo apt-get clean' 2021-03-02T23:59:24.415 INFO:scrape:2 jobs: ['5928401', '5928400'] 2021-03-02T23:59:24.415 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'osd}', 'overrides/{frag_enable', 'overrides}}', 'session_timeout', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-02T23:59:24.416 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'overrides}}', 'ranks/1', 'ranks/5', 'session_timeout', 'tasks/dbench', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-02T23:59:24.416 INFO:scrape: 2021-03-02T23:59:24.416 INFO:scrape:Failure: Stale jobs detected, aborting. 2021-03-02T23:59:24.417 INFO:scrape:1 jobs: ['5928398'] 2021-03-02T23:59:24.417 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag_enable', 'overrides}}', 'ranks/1', 'session_timeout', 'tasks/dbench', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-02T23:59:24.418 INFO:scrape: 2021-03-02T23:59:24.418 INFO:scrape:Failure: {'Failure object was': {'smithi105.front.sepia.ceph.com': {'changed': True, 'end': '2021-03-02 23:21:35.816685', 'stdout': '', 'cmd': ['sudo', 'apt-get', 'clean'], 'delta': '0:00:00.060983', 'stderr': 'E: Could not get lock /var/cache/apt/archives/lock. It is held by process 4071 (apt-get)\\nE: Unable to lock directory /var/cache/apt/archives/', 'rc': 100, 'invocation': {'module_args': {'creates': 'None', 'executable': 'None', '_uses_shell': False, 'strip_empty_ends': True, '_raw_params': 'sudo apt-get clean', 'removes': 'None', 'argv': 'None', 'warn': True, 'chdir': 'None', 'stdin_add_newline': True, 'stdin': 'None'}}, 'start': '2021-03-02 23:21:35.755702', 'warnings': ["Consider using 'become', 'become_method', and 'become_user' rather than running sudo"], 'msg': 'non-zero return code', 'stdout_lines': [], 'stderr_lines': ['E: Could not get lock /var/cache/apt/archives/lock. It is held by process 4071 (apt-get)', 'E: Unable to lock directory /var/cache/apt/archives/'], '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'changed')"} 2021-03-02T23:59:24.419 INFO:scrape:1 jobs: ['5928397'] 2021-03-02T23:59:24.419 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'overrides}}', 'ranks/1', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-02T23:59:24.419 INFO:scrape: 2021-03-02T23:59:24.421 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/ failed: 8 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [5928396] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:37:58 info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928396/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928396/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63a1fe1ec325479c84bca61356d079dc saw valgrind issues [5928397] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:02:12 info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928397/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=637bd209e0bf48e6bd93654d5d768350 {'Failure object was': {'smithi105.front.sepia.ceph.com': {'changed': True, 'end': '2021-03-02 23:21:35.816685', 'stdout': '', 'cmd': ['sudo', 'apt- get', 'clean'], 'delta': '0:00:00.060983', 'stderr': 'E: Could not get lock /var/cache/apt/archives/lock. It is held by process 4071 (apt-get)\\nE: Unable to lock directory /var/cache/apt/archives/', 'rc': 100, 'invocation': {'module_args': {'creates': 'None', 'executable': 'None', '_uses_shell': False, 'strip_empty_ends': True, '_raw_params': 'sudo apt- get clean', 'removes': 'None', 'argv': 'None', 'warn': True, 'chdir': 'None', 'stdin_add_newline': True, 'stdin': 'None'}}, 'start': '2021-03-02 23:21:35.755702', 'warnings': ["Consider using 'become', 'become_method', and 'become_user' rather than running sudo"], 'msg': 'non-zero return code', 'stdout_lines': [], 'stderr_lines': ['E: Could not get lock /var/cache/apt/archives/lock. It is held by process 4071 (apt-get)', 'E: Unable to lock directory /var/cache/apt/archives/'], '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74f cd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74f cd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'changed')"} [5928398] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928398/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928398/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b5e6bd061694ff3852e3c12b283c6da Stale jobs detected, aborting. [5928399] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:33:05 info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928399/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928399/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53d7206493fc4abe91aed0c8495a3fca saw valgrind issues [5928400] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:32:43 info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928400/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928400/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83dca7718a0f42b884bee5e3e494fa87 Command failed on smithi198 with status 1: 'set -ex\nsudo dd of=/home/ubuntu/cephtest/valgrind.supp' [5928401] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928401/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a324257aaf64ad2afa8dbc507652e30 Command failed on smithi039 with status 100: 'sudo apt-get clean' [5928402] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:39:31 info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928402/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=522fc4155e414358aefcd7cc7a571e1d object of type 'generator' has no len() [5928403] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:37:49 info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928403/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928403/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eba3d2adf6004f00a465baf86256a00e saw valgrind issues Dead ================================================================= [5928404] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:02:47 info: http://pulpito.ceph.com/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928404/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_23:11:25-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5928404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d5c680913ae4c53b24ffdb989026eb1 {'smithi203.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi203.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi203.front.sepia.ceph.com port 22: No route to host\r\n', 'unreachable': True}}