2023-05-17T11:53:02.180 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-17T11:53:02.181 INFO:teuthology.results:subset: '344/1024' 2023-05-17T11:53:02.182 INFO:teuthology.results:seed: '1357' 2023-05-17T13:01:05.739 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-17T13:01:05.768 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-18T01:01:57.591 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-05-18T01:02:01.184 INFO:scrape:Found 42 jobs 2023-05-18T01:02:07.184 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-3942-g280675bb (280675bb3d0dbcfdc8e64b09f2328a7c957e9ccb) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56090fe2edaf] , 2: (KernelDevice::_aio_thread()+0x1798) [0x560910a41fc8] 2023-05-18T01:02:07.185 WARNING:scrape:Ignoring malparsed backtrace: nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 , ceph-fuse /tmp/tmp.3n61ha2sIstest fuse.ceph-fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0 , 2023-05-18T01:02:07.185 WARNING:scrape:Ignoring malparsed backtrace: nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 , nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 , 2023-05-18T01:02:07.186 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T17:11:02.126+0000 7f11165f9700 -1 osd.4 46 *** Immediate shutdown (osd_fast_shutdown=true) *** , 2023-05-17T17:11:02.125+0000 7f4e2a7f9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i d (PID: 112321) UID: 0 , 2023-05-17T17:11:02.125+0000 7f4e2a7f9700 -1 mds.d *** got signal Terminated *** 2023-05-18T01:02:07.186 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T17:33:24.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T17:33:24.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T17:33:54.611 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.187 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T17:58:36.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T17:59:06.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T17:59:06.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.187 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T18:24:18.749 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T18:24:18.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T18:24:49.000 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.187 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T18:49:30.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T18:50:01.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T18:50:01.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.188 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T19:15:12.798 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T19:15:12.806 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T19:15:43.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.188 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T19:40:24.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T19:40:54.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T19:40:54.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.189 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T20:06:06.295 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T20:06:06.304 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T20:06:36.506 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.189 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T20:31:17.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T20:31:47.664 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T20:31:47.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.190 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T20:56:58.648 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T20:56:58.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T20:57:28.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.190 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T21:22:09.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T21:22:39.546 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T21:22:39.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.190 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T21:47:51.343 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T21:47:51.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T21:48:21.581 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.191 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T22:13:02.937 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T22:13:33.133 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T22:13:33.140 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.191 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T22:38:44.686 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T22:38:44.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T22:39:14.972 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.191 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T23:03:56.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T23:04:26.516 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T23:04:26.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.192 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-17T23:29:37.677 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T23:29:37.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-17T23:30:07.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-18T01:02:07.203 WARNING:scrape:Not checking for backtrace from smithi088:osd.2 log, too large (500920560) 2023-05-18T01:02:07.243 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/results.py", line 71, in results Scraper(archive_dir).analyze() File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/scrape.py", line 491, in analyze if reason.match(job): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/scrape.py", line 194, in match if job.get_backtrace(): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/scrape.py", line 339, in get_backtrace self._populate_backtrace() File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/scrape.py", line 385, in _populate_backtrace bt, ass = self._search_backtrace(GzipFile(gzipped_log_path)) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/scrape.py", line 305, in _search_backtrace if ".stderr:" in line: TypeError: a bytes-like object is required, not 'str'