2021-02-05T02:44:46.089 INFO:root:teuthology version: 1.0.0-5251f0da 2021-02-05T02:44:46.091 INFO:teuthology.results:subset: '9/99999' 2021-02-05T02:44:46.091 INFO:teuthology.results:seed: '6067' 2021-02-05T07:48:28.159 INFO:root:teuthology version: 1.0.0-5251f0da 2021-02-05T07:48:28.229 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-02-05T17:29:50.051 INFO:teuthology.results:Tests finished! gathering results... 2021-02-05T17:32:21.944 INFO:scrape:Found 378 jobs 2021-02-05T17:32:21.952 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2021-02-03_03:31:04-rados-pacific-distro-basic-smithi/5852258/teuthology.log 2021-02-05T17:32:38.026 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.1.0-100-g74275226 (74275226ac79999bfd40e683dc9a1309e76033bf) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf09d6a15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55b993c43f36] 2021-02-05T17:32:38.028 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-02-05T06:48:11.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-02-05T06:48:11.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2021-02-05T17:32:38.028 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:49:13.421 INFO:tasks.ceph.mon.a:Stopped , 2021-02-05T06:49:13.422 INFO:tasks.ceph:Checking cluster log for badness... , 2021-02-05T06:49:13.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-02-05T17:32:38.029 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:51:30.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-02-05T06:51:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-02-05T06:51:30.242 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-05T17:32:38.136 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.1.0-100-g74275226 (74275226ac79999bfd40e683dc9a1309e76033bf) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf09d6a15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55b993c43f36] 2021-02-05T17:32:38.137 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-02-05T06:48:11.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-02-05T06:48:11.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2021-02-05T17:32:38.138 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:49:13.421 INFO:tasks.ceph.mon.a:Stopped , 2021-02-05T06:49:13.422 INFO:tasks.ceph:Checking cluster log for badness... , 2021-02-05T06:49:13.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-02-05T17:32:38.139 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:51:30.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-02-05T06:51:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-02-05T06:51:30.242 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-05T17:32:38.241 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.1.0-100-g74275226 (74275226ac79999bfd40e683dc9a1309e76033bf) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf09d6a15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55b993c43f36] 2021-02-05T17:32:38.243 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-02-05T06:48:11.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-02-05T06:48:11.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2021-02-05T17:32:38.244 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:49:13.421 INFO:tasks.ceph.mon.a:Stopped , 2021-02-05T06:49:13.422 INFO:tasks.ceph:Checking cluster log for badness... , 2021-02-05T06:49:13.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-02-05T17:32:38.244 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:51:30.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-02-05T06:51:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-02-05T06:51:30.242 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-05T17:32:38.352 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.1.0-100-g74275226 (74275226ac79999bfd40e683dc9a1309e76033bf) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf09d6a15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55b993c43f36] 2021-02-05T17:32:38.354 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-02-05T06:48:11.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-02-05T06:48:11.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2021-02-05T17:32:38.354 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:49:13.421 INFO:tasks.ceph.mon.a:Stopped , 2021-02-05T06:49:13.422 INFO:tasks.ceph:Checking cluster log for badness... , 2021-02-05T06:49:13.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-02-05T17:32:38.355 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:51:30.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-02-05T06:51:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-02-05T06:51:30.242 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-05T17:32:38.465 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.1.0-100-g74275226 (74275226ac79999bfd40e683dc9a1309e76033bf) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf09d6a15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55b993c43f36] 2021-02-05T17:32:38.467 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-02-05T06:48:11.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-02-05T06:48:11.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2021-02-05T17:32:38.468 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:49:13.421 INFO:tasks.ceph.mon.a:Stopped , 2021-02-05T06:49:13.422 INFO:tasks.ceph:Checking cluster log for badness... , 2021-02-05T06:49:13.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-02-05T17:32:38.468 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:51:30.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-02-05T06:51:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-02-05T06:51:30.242 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-05T17:32:38.570 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.1.0-100-g74275226 (74275226ac79999bfd40e683dc9a1309e76033bf) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf09d6a15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55b993c43f36] 2021-02-05T17:32:38.572 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-02-05T06:48:11.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-02-05T06:48:11.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2021-02-05T17:32:38.573 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:49:13.421 INFO:tasks.ceph.mon.a:Stopped , 2021-02-05T06:49:13.422 INFO:tasks.ceph:Checking cluster log for badness... , 2021-02-05T06:49:13.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-02-05T17:32:38.573 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-05T06:51:30.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-02-05T06:51:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-02-05T06:51:30.242 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-05T17:32:39.453 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_5251f0da07972a7365332a5ff74fa4d328580822/teuthology/results.py", line 34, in main int(args['--timeout']), args['--dry-run']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_5251f0da07972a7365332a5ff74fa4d328580822/teuthology/results.py", line 69, in results Scraper(archive_dir).analyze() File "/home/teuthworker/src/git.ceph.com_git_teuthology_5251f0da07972a7365332a5ff74fa4d328580822/teuthology/scrape.py", line 490, in analyze if reason.match(job): File "/home/teuthworker/src/git.ceph.com_git_teuthology_5251f0da07972a7365332a5ff74fa4d328580822/teuthology/scrape.py", line 204, in match job.get_last_tlog_line()).ratio() File "/usr/lib/python3.6/difflib.py", line 644, in ratio matches = sum(triple[-1] for triple in self.get_matching_blocks()) File "/usr/lib/python3.6/difflib.py", line 467, in get_matching_blocks la, lb = len(self.a), len(self.b) TypeError: object of type 'NoneType' has no len()