2021-01-28T00:17:25.886 INFO:root:teuthology version: 1.0.0-83ba941a 2021-01-28T00:17:25.889 INFO:teuthology.results:subset: '1/20' 2021-01-28T00:17:25.890 INFO:teuthology.results:seed: '5385' 2021-01-28T00:29:01.983 INFO:root:teuthology version: 1.0.0-83ba941a 2021-01-28T00:29:02.068 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-28T01:55:03.622 INFO:teuthology.results:Tests finished! gathering results... 2021-01-28T01:55:04.217 INFO:scrape:Found 5 jobs 2021-01-28T01:55:04.819 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-189-g3558fd72 (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f142e2f0a1a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c34) [0x7f142e2f0c34] 2021-01-28T01:55:04.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:53:10.204 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 , instructed osd(s) 5 to deep-scrub , 2021-01-28T00:53:10.514 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-28T01:55:04.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:54:12.601 INFO:tasks.ceph.mon.a:Stopped , 2021-01-28T00:54:12.602 INFO:tasks.ceph:Checking cluster log for badness... , 2021-01-28T00:54:12.602 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-01-28T01:55:04.822 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:57:19.058 DEBUG:teuthology.orchestra.run.smithi093:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-28T00:57:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-01-28T00:57:19.090 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-01-28T01:55:04.985 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-189-g3558fd72 (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f142e2f0a1a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c34) [0x7f142e2f0c34] 2021-01-28T01:55:04.987 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:53:10.204 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 , instructed osd(s) 5 to deep-scrub , 2021-01-28T00:53:10.514 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-28T01:55:04.988 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:54:12.601 INFO:tasks.ceph.mon.a:Stopped , 2021-01-28T00:54:12.602 INFO:tasks.ceph:Checking cluster log for badness... , 2021-01-28T00:54:12.602 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-01-28T01:55:04.988 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:57:19.058 DEBUG:teuthology.orchestra.run.smithi093:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-28T00:57:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-01-28T00:57:19.090 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-01-28T01:55:05.143 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-189-g3558fd72 (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f142e2f0a1a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c34) [0x7f142e2f0c34] 2021-01-28T01:55:05.145 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:53:10.204 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 , instructed osd(s) 5 to deep-scrub , 2021-01-28T00:53:10.514 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-28T01:55:05.145 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:54:12.601 INFO:tasks.ceph.mon.a:Stopped , 2021-01-28T00:54:12.602 INFO:tasks.ceph:Checking cluster log for badness... , 2021-01-28T00:54:12.602 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-01-28T01:55:05.146 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:57:19.058 DEBUG:teuthology.orchestra.run.smithi093:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-28T00:57:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-01-28T00:57:19.090 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-01-28T01:55:05.480 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-189-g3558fd72 (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f8c6f9d2a1a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c34) [0x7f8c6f9d2c34] 2021-01-28T01:55:05.482 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T01:19:11.851 DEBUG:teuthology.orchestra.run.smithi201:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T01:19:12.141 INFO:teuthology.orchestra.run.smithi201.stdout: 2021-01-28T01:55:05.482 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T01:20:13.872 INFO:tasks.ceph.mon.a:Stopped , 2021-01-28T01:20:13.873 INFO:tasks.ceph:Checking cluster log for badness... , 2021-01-28T01:20:13.873 DEBUG:teuthology.orchestra.run.smithi201:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-01-28T01:55:05.483 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T01:21:00.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-28T01:21:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-01-28T01:21:00.889 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-01-28T01:55:05.805 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-189-g3558fd72 (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7fa5437fb15a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x559543d04914] 2021-01-28T01:55:05.807 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T00:56:20.029 DEBUG:teuthology.orchestra.run.smithi201:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T00:56:20.397 INFO:teuthology.orchestra.run.smithi201.stdout: 2021-01-28T01:55:05.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:57:22.161 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-01-28T00:57:22.161 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-28T00:57:22.209 INFO:tasks.ceph.mon.a:Stopped 2021-01-28T01:55:05.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T00:59:14.803 DEBUG:teuthology.orchestra.run.smithi201:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-28T00:59:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-01-28T00:59:14.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-01-28T01:55:06.300 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-189-g3558fd72 (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff807ccaa1a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c34) [0x7ff807ccac34] 2021-01-28T01:55:06.302 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T01:47:37.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T01:47:37.315 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-01-28T01:55:06.303 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T01:48:59.470 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-28T01:48:59.489 INFO:tasks.ceph.mgr.x:Stopped , 2021-01-28T01:48:59.490 INFO:teuthology.misc:Shutting down mon daemons... 2021-01-28T01:55:06.303 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T01:53:37.698 DEBUG:teuthology.run_tasks:Unwinding manager selinux , 2021-01-28T01:53:37.713 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log , 2021-01-28T01:53:37.775 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-01-28T01:55:06.701 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-189-g3558fd72 (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f3c5de74a1a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c34) [0x7f3c5de74c34] 2021-01-28T01:55:06.703 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T01:22:16.849 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 , instructed osd(s) 5 to deep-scrub , 2021-01-28T01:22:17.210 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-28T01:55:06.703 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T01:23:39.345 INFO:tasks.ceph.osd.4:Stopped , 2021-01-28T01:23:39.346 DEBUG:tasks.ceph.osd.5:waiting for process to exit , 2021-01-28T01:23:39.346 INFO:teuthology.orchestra.run:waiting for 300 2021-01-28T01:55:06.704 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-28T01:25:41.534 INFO:teuthology.orchestra.run.smithi067.stdout:^- hv01.front.sepia.ceph.com 3 6 377 40 +1252us[+1253us] +/- 90ms , 2021-01-28T01:25:41.534 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 2 6 377 39 -515us[ -514us] +/- 42ms , 2021-01-28T01:25:41.535 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +283us[ +283us] +/- 84ms 2021-01-28T01:55:06.720 INFO:scrape:Found 1 distinct failure reasons 2021-01-28T01:55:06.721 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-01-28T01:55:06.721 INFO:scrape:5 jobs: ['5834007', '5834006', '5834005', '5834004', '5834008'] 2021-01-28T01:55:06.722 INFO:scrape:suites intersection: ['rados', 'rados:singleton/{all/thrash_cache_writeback_proxy_none'] 2021-01-28T01:55:06.722 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'rados', 'rados:singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}'] 2021-01-28T01:55:06.722 INFO:scrape: