2020-06-29T08:24:25.833 INFO:root:teuthology version: 1.0.0-22bcea5 2020-06-29T08:24:25.835 INFO:teuthology.results:subset: '268/9999' 2020-06-29T08:24:25.835 INFO:teuthology.results:seed: '1501' 2020-06-29T08:36:23.483 INFO:root:teuthology version: 1.0.0-22bcea5 2020-06-29T08:36:23.486 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-06-29T20:34:37.793 INFO:teuthology.results:Tests finished! gathering results... 2020-06-29T20:34:39.900 INFO:scrape:Found 21 jobs 2020-06-29T20:34:43.871 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-2924-g3a96153 (3a961532ad4058431d46b1d6573ce0aff5065bea) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff02001e9ea] , 2: (()+0x28bc04) [0x7ff02001ec04] 2020-06-29T20:34:43.875 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-06-29T09:00:00.740 INFO:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-06-29T09:00:01.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-06-29T20:34:43.876 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:01:22.905 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-06-29T09:01:22.906 INFO:teuthology.orchestra.run:waiting for 300 , 2020-06-29T09:01:22.921 INFO:tasks.ceph.mgr.x:Stopped 2020-06-29T20:34:43.880 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:16.0.0-2924.g3a96153.el8 @ceph 40 M , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-06-29T20:34:43.881 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:50.394 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.452 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.534 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : socat-1.7.3.2-6.el8.x86_64 4/5 2020-06-29T20:34:43.882 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:16.0.0-2924.g3a96153.el8.x86_64 1/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:16.0.0-2924.g3a96153.el8.x86_64 2/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:16.0.0-2924.g3a96153.el8.x86_64 3/4 2020-06-29T20:34:43.883 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch0.8.1-17.el8 @epel 24 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch3.4.0.3-18.el8 @epel 43 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch1.14.1-5.el8 @epel 82 k 2020-06-29T20:34:43.884 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch1.24.2-2.el8 @CentOS-Base 604 k , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch1.2.1-2.el8.1 @epel 1.0 M , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings noarch0.5.1-6.el8 @CentOS-AppStream 72 k 2020-06-29T20:34:43.885 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:01.258 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-portend-2.6-1.el8.noarch 26/85 , 2020-06-29T09:03:01.330 INFO:teuthology.orchestra.run.smithi176.stdout: Erasing : python3-rsa-3.4.2-12.el8.noarch 30/85 , 2020-06-29T09:03:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-tempora-1.14.1-5.el8.noarch 27/85 2020-06-29T20:34:43.886 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2020-06-29T20:34:43.888 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 45/85 , 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 46/85 , 2020-06-29T09:03:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 47/85 2020-06-29T20:34:43.891 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-11.0.0-2.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-29T20:34:43.892 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 61/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 62/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:16.0.0-2924.g3a96153.el8.x86_64 63/85 2020-06-29T20:34:43.892 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 , 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch , 2020-06-29T09:03:29.766 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-29T20:34:43.893 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. , 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! , 2020-06-29T09:03:32.847 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-06-29T20:34:43.894 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout:Removed: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.0.0-2924.g3a96153.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-29T20:34:43.894 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:39.036 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-06-29T20:34:43.895 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:Removing: 2020-06-29T20:34:43.896 WARNING:scrape:Ignoring malparsed backtrace: No packages marked for removal. , 2020-06-29T09:03:45.975 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. , 2020-06-29T09:03:45.976 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2020-06-29T20:34:43.896 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:50.349 INFO:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , 2020-06-29T09:03:50.356 INFO:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service 2020-06-29T20:34:44.044 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-2924-g3a96153 (3a961532ad4058431d46b1d6573ce0aff5065bea) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff02001e9ea] , 2: (()+0x28bc04) [0x7ff02001ec04] 2020-06-29T20:34:44.047 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-06-29T09:00:00.740 INFO:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-06-29T09:00:01.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-06-29T20:34:44.048 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:01:22.905 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-06-29T09:01:22.906 INFO:teuthology.orchestra.run:waiting for 300 , 2020-06-29T09:01:22.921 INFO:tasks.ceph.mgr.x:Stopped 2020-06-29T20:34:44.049 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:16.0.0-2924.g3a96153.el8 @ceph 40 M , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-06-29T20:34:44.050 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:50.394 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.452 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.534 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : socat-1.7.3.2-6.el8.x86_64 4/5 2020-06-29T20:34:44.050 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:16.0.0-2924.g3a96153.el8.x86_64 1/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:16.0.0-2924.g3a96153.el8.x86_64 2/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:16.0.0-2924.g3a96153.el8.x86_64 3/4 2020-06-29T20:34:44.051 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch0.8.1-17.el8 @epel 24 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch3.4.0.3-18.el8 @epel 43 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch1.14.1-5.el8 @epel 82 k 2020-06-29T20:34:44.051 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch1.24.2-2.el8 @CentOS-Base 604 k , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch1.2.1-2.el8.1 @epel 1.0 M , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings noarch0.5.1-6.el8 @CentOS-AppStream 72 k 2020-06-29T20:34:44.052 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:01.258 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-portend-2.6-1.el8.noarch 26/85 , 2020-06-29T09:03:01.330 INFO:teuthology.orchestra.run.smithi176.stdout: Erasing : python3-rsa-3.4.2-12.el8.noarch 30/85 , 2020-06-29T09:03:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-tempora-1.14.1-5.el8.noarch 27/85 2020-06-29T20:34:44.053 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2020-06-29T20:34:44.053 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 45/85 , 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 46/85 , 2020-06-29T09:03:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 47/85 2020-06-29T20:34:44.054 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-11.0.0-2.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-29T20:34:44.055 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 61/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 62/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:16.0.0-2924.g3a96153.el8.x86_64 63/85 2020-06-29T20:34:44.055 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 , 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch , 2020-06-29T09:03:29.766 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-29T20:34:44.056 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. , 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! , 2020-06-29T09:03:32.847 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-06-29T20:34:44.056 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout:Removed: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.0.0-2924.g3a96153.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-29T20:34:44.057 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:39.036 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-06-29T20:34:44.058 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:Removing: 2020-06-29T20:34:44.058 WARNING:scrape:Ignoring malparsed backtrace: No packages marked for removal. , 2020-06-29T09:03:45.975 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. , 2020-06-29T09:03:45.976 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2020-06-29T20:34:44.059 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:50.349 INFO:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , 2020-06-29T09:03:50.356 INFO:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service 2020-06-29T20:34:44.203 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-2924-g3a96153 (3a961532ad4058431d46b1d6573ce0aff5065bea) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff02001e9ea] , 2: (()+0x28bc04) [0x7ff02001ec04] 2020-06-29T20:34:44.205 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-06-29T09:00:00.740 INFO:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-06-29T09:00:01.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-06-29T20:34:44.206 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:01:22.905 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-06-29T09:01:22.906 INFO:teuthology.orchestra.run:waiting for 300 , 2020-06-29T09:01:22.921 INFO:tasks.ceph.mgr.x:Stopped 2020-06-29T20:34:44.207 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:16.0.0-2924.g3a96153.el8 @ceph 40 M , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-06-29T20:34:44.207 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:50.394 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.452 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.534 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : socat-1.7.3.2-6.el8.x86_64 4/5 2020-06-29T20:34:44.208 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:16.0.0-2924.g3a96153.el8.x86_64 1/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:16.0.0-2924.g3a96153.el8.x86_64 2/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:16.0.0-2924.g3a96153.el8.x86_64 3/4 2020-06-29T20:34:44.209 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch0.8.1-17.el8 @epel 24 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch3.4.0.3-18.el8 @epel 43 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch1.14.1-5.el8 @epel 82 k 2020-06-29T20:34:44.209 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch1.24.2-2.el8 @CentOS-Base 604 k , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch1.2.1-2.el8.1 @epel 1.0 M , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings noarch0.5.1-6.el8 @CentOS-AppStream 72 k 2020-06-29T20:34:44.210 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:01.258 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-portend-2.6-1.el8.noarch 26/85 , 2020-06-29T09:03:01.330 INFO:teuthology.orchestra.run.smithi176.stdout: Erasing : python3-rsa-3.4.2-12.el8.noarch 30/85 , 2020-06-29T09:03:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-tempora-1.14.1-5.el8.noarch 27/85 2020-06-29T20:34:44.210 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2020-06-29T20:34:44.211 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 45/85 , 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 46/85 , 2020-06-29T09:03:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 47/85 2020-06-29T20:34:44.212 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-11.0.0-2.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-29T20:34:44.212 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 61/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 62/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:16.0.0-2924.g3a96153.el8.x86_64 63/85 2020-06-29T20:34:44.213 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 , 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch , 2020-06-29T09:03:29.766 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-29T20:34:44.213 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. , 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! , 2020-06-29T09:03:32.847 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-06-29T20:34:44.214 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout:Removed: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.0.0-2924.g3a96153.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-29T20:34:44.215 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:39.036 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-06-29T20:34:44.215 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:Removing: 2020-06-29T20:34:44.216 WARNING:scrape:Ignoring malparsed backtrace: No packages marked for removal. , 2020-06-29T09:03:45.975 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. , 2020-06-29T09:03:45.976 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2020-06-29T20:34:44.216 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:50.349 INFO:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , 2020-06-29T09:03:50.356 INFO:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service 2020-06-29T20:34:44.352 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-2924-g3a96153 (3a961532ad4058431d46b1d6573ce0aff5065bea) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff02001e9ea] , 2: (()+0x28bc04) [0x7ff02001ec04] 2020-06-29T20:34:44.355 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-06-29T09:00:00.740 INFO:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-06-29T09:00:01.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-06-29T20:34:44.355 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:01:22.905 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-06-29T09:01:22.906 INFO:teuthology.orchestra.run:waiting for 300 , 2020-06-29T09:01:22.921 INFO:tasks.ceph.mgr.x:Stopped 2020-06-29T20:34:44.356 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:16.0.0-2924.g3a96153.el8 @ceph 40 M , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-06-29T20:34:44.357 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:50.394 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.452 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.534 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : socat-1.7.3.2-6.el8.x86_64 4/5 2020-06-29T20:34:44.357 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:16.0.0-2924.g3a96153.el8.x86_64 1/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:16.0.0-2924.g3a96153.el8.x86_64 2/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:16.0.0-2924.g3a96153.el8.x86_64 3/4 2020-06-29T20:34:44.358 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch0.8.1-17.el8 @epel 24 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch3.4.0.3-18.el8 @epel 43 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch1.14.1-5.el8 @epel 82 k 2020-06-29T20:34:44.359 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch1.24.2-2.el8 @CentOS-Base 604 k , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch1.2.1-2.el8.1 @epel 1.0 M , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings noarch0.5.1-6.el8 @CentOS-AppStream 72 k 2020-06-29T20:34:44.359 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:01.258 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-portend-2.6-1.el8.noarch 26/85 , 2020-06-29T09:03:01.330 INFO:teuthology.orchestra.run.smithi176.stdout: Erasing : python3-rsa-3.4.2-12.el8.noarch 30/85 , 2020-06-29T09:03:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-tempora-1.14.1-5.el8.noarch 27/85 2020-06-29T20:34:44.360 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2020-06-29T20:34:44.360 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 45/85 , 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 46/85 , 2020-06-29T09:03:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 47/85 2020-06-29T20:34:44.361 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-11.0.0-2.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-29T20:34:44.362 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 61/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 62/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:16.0.0-2924.g3a96153.el8.x86_64 63/85 2020-06-29T20:34:44.362 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 , 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch , 2020-06-29T09:03:29.766 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-29T20:34:44.363 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. , 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! , 2020-06-29T09:03:32.847 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-06-29T20:34:44.364 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout:Removed: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.0.0-2924.g3a96153.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-29T20:34:44.364 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:39.036 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-06-29T20:34:44.365 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:Removing: 2020-06-29T20:34:44.365 WARNING:scrape:Ignoring malparsed backtrace: No packages marked for removal. , 2020-06-29T09:03:45.975 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. , 2020-06-29T09:03:45.976 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2020-06-29T20:34:44.366 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:50.349 INFO:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , 2020-06-29T09:03:50.356 INFO:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service 2020-06-29T20:34:44.509 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-2924-g3a96153 (3a961532ad4058431d46b1d6573ce0aff5065bea) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff02001e9ea] , 2: (()+0x28bc04) [0x7ff02001ec04] 2020-06-29T20:34:44.511 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-06-29T09:00:00.740 INFO:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-06-29T09:00:01.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-06-29T20:34:44.512 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:01:22.905 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-06-29T09:01:22.906 INFO:teuthology.orchestra.run:waiting for 300 , 2020-06-29T09:01:22.921 INFO:tasks.ceph.mgr.x:Stopped 2020-06-29T20:34:44.513 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:16.0.0-2924.g3a96153.el8 @ceph 40 M , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-06-29T20:34:44.513 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:50.394 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.452 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.534 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : socat-1.7.3.2-6.el8.x86_64 4/5 2020-06-29T20:34:44.514 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:16.0.0-2924.g3a96153.el8.x86_64 1/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:16.0.0-2924.g3a96153.el8.x86_64 2/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:16.0.0-2924.g3a96153.el8.x86_64 3/4 2020-06-29T20:34:44.515 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch0.8.1-17.el8 @epel 24 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch3.4.0.3-18.el8 @epel 43 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch1.14.1-5.el8 @epel 82 k 2020-06-29T20:34:44.515 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch1.24.2-2.el8 @CentOS-Base 604 k , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch1.2.1-2.el8.1 @epel 1.0 M , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings noarch0.5.1-6.el8 @CentOS-AppStream 72 k 2020-06-29T20:34:44.516 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:01.258 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-portend-2.6-1.el8.noarch 26/85 , 2020-06-29T09:03:01.330 INFO:teuthology.orchestra.run.smithi176.stdout: Erasing : python3-rsa-3.4.2-12.el8.noarch 30/85 , 2020-06-29T09:03:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-tempora-1.14.1-5.el8.noarch 27/85 2020-06-29T20:34:44.517 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2020-06-29T20:34:44.517 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 45/85 , 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 46/85 , 2020-06-29T09:03:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 47/85 2020-06-29T20:34:44.518 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-11.0.0-2.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-29T20:34:44.519 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 61/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 62/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:16.0.0-2924.g3a96153.el8.x86_64 63/85 2020-06-29T20:34:44.519 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 , 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch , 2020-06-29T09:03:29.766 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-29T20:34:44.520 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. , 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! , 2020-06-29T09:03:32.847 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-06-29T20:34:44.520 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout:Removed: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.0.0-2924.g3a96153.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-29T20:34:44.521 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:39.036 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-06-29T20:34:44.522 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:Removing: 2020-06-29T20:34:44.522 WARNING:scrape:Ignoring malparsed backtrace: No packages marked for removal. , 2020-06-29T09:03:45.975 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. , 2020-06-29T09:03:45.976 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2020-06-29T20:34:44.523 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:50.349 INFO:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , 2020-06-29T09:03:50.356 INFO:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service 2020-06-29T20:34:44.660 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-2924-g3a96153 (3a961532ad4058431d46b1d6573ce0aff5065bea) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff02001e9ea] , 2: (()+0x28bc04) [0x7ff02001ec04] 2020-06-29T20:34:44.663 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-06-29T09:00:00.740 INFO:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-06-29T09:00:01.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-06-29T20:34:44.664 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:01:22.905 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-06-29T09:01:22.906 INFO:teuthology.orchestra.run:waiting for 300 , 2020-06-29T09:01:22.921 INFO:tasks.ceph.mgr.x:Stopped 2020-06-29T20:34:44.665 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:16.0.0-2924.g3a96153.el8 @ceph 40 M , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-06-29T20:34:44.665 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:50.394 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.452 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.534 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : socat-1.7.3.2-6.el8.x86_64 4/5 2020-06-29T20:34:44.666 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:16.0.0-2924.g3a96153.el8.x86_64 1/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:16.0.0-2924.g3a96153.el8.x86_64 2/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:16.0.0-2924.g3a96153.el8.x86_64 3/4 2020-06-29T20:34:44.667 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch0.8.1-17.el8 @epel 24 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch3.4.0.3-18.el8 @epel 43 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch1.14.1-5.el8 @epel 82 k 2020-06-29T20:34:44.668 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch1.24.2-2.el8 @CentOS-Base 604 k , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch1.2.1-2.el8.1 @epel 1.0 M , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings noarch0.5.1-6.el8 @CentOS-AppStream 72 k 2020-06-29T20:34:44.668 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:01.258 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-portend-2.6-1.el8.noarch 26/85 , 2020-06-29T09:03:01.330 INFO:teuthology.orchestra.run.smithi176.stdout: Erasing : python3-rsa-3.4.2-12.el8.noarch 30/85 , 2020-06-29T09:03:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-tempora-1.14.1-5.el8.noarch 27/85 2020-06-29T20:34:44.670 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2020-06-29T20:34:44.671 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 45/85 , 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 46/85 , 2020-06-29T09:03:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 47/85 2020-06-29T20:34:44.672 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-11.0.0-2.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-29T20:34:44.672 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 61/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 62/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:16.0.0-2924.g3a96153.el8.x86_64 63/85 2020-06-29T20:34:44.673 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 , 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch , 2020-06-29T09:03:29.766 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-29T20:34:44.674 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. , 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! , 2020-06-29T09:03:32.847 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-06-29T20:34:44.674 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout:Removed: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.0.0-2924.g3a96153.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-29T20:34:44.675 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:39.036 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-06-29T20:34:44.675 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:Removing: 2020-06-29T20:34:44.676 WARNING:scrape:Ignoring malparsed backtrace: No packages marked for removal. , 2020-06-29T09:03:45.975 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. , 2020-06-29T09:03:45.976 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2020-06-29T20:34:44.677 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:50.349 INFO:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , 2020-06-29T09:03:50.356 INFO:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service 2020-06-29T20:34:44.814 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-2924-g3a96153 (3a961532ad4058431d46b1d6573ce0aff5065bea) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff02001e9ea] , 2: (()+0x28bc04) [0x7ff02001ec04] 2020-06-29T20:34:44.817 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-06-29T09:00:00.740 INFO:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-06-29T09:00:01.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-06-29T20:34:44.818 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:01:22.905 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-06-29T09:01:22.906 INFO:teuthology.orchestra.run:waiting for 300 , 2020-06-29T09:01:22.921 INFO:tasks.ceph.mgr.x:Stopped 2020-06-29T20:34:44.818 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:16.0.0-2924.g3a96153.el8 @ceph 40 M , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-06-29T20:34:44.819 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:50.394 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.452 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.534 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : socat-1.7.3.2-6.el8.x86_64 4/5 2020-06-29T20:34:44.820 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:16.0.0-2924.g3a96153.el8.x86_64 1/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:16.0.0-2924.g3a96153.el8.x86_64 2/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:16.0.0-2924.g3a96153.el8.x86_64 3/4 2020-06-29T20:34:44.820 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch0.8.1-17.el8 @epel 24 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch3.4.0.3-18.el8 @epel 43 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch1.14.1-5.el8 @epel 82 k 2020-06-29T20:34:44.821 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch1.24.2-2.el8 @CentOS-Base 604 k , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch1.2.1-2.el8.1 @epel 1.0 M , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings noarch0.5.1-6.el8 @CentOS-AppStream 72 k 2020-06-29T20:34:44.822 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:01.258 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-portend-2.6-1.el8.noarch 26/85 , 2020-06-29T09:03:01.330 INFO:teuthology.orchestra.run.smithi176.stdout: Erasing : python3-rsa-3.4.2-12.el8.noarch 30/85 , 2020-06-29T09:03:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-tempora-1.14.1-5.el8.noarch 27/85 2020-06-29T20:34:44.822 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2020-06-29T20:34:44.823 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 45/85 , 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 46/85 , 2020-06-29T09:03:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 47/85 2020-06-29T20:34:44.824 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-11.0.0-2.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-29T20:34:44.824 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 61/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 62/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:16.0.0-2924.g3a96153.el8.x86_64 63/85 2020-06-29T20:34:44.825 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 , 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch , 2020-06-29T09:03:29.766 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-29T20:34:44.825 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. , 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! , 2020-06-29T09:03:32.847 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-06-29T20:34:44.826 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout:Removed: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.0.0-2924.g3a96153.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-29T20:34:44.827 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:39.036 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-06-29T20:34:44.827 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:Removing: 2020-06-29T20:34:44.828 WARNING:scrape:Ignoring malparsed backtrace: No packages marked for removal. , 2020-06-29T09:03:45.975 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. , 2020-06-29T09:03:45.976 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2020-06-29T20:34:44.829 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:50.349 INFO:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , 2020-06-29T09:03:50.356 INFO:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service 2020-06-29T20:34:44.968 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-2924-g3a96153 (3a961532ad4058431d46b1d6573ce0aff5065bea) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff02001e9ea] , 2: (()+0x28bc04) [0x7ff02001ec04] 2020-06-29T20:34:44.971 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-06-29T09:00:00.740 INFO:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-06-29T09:00:01.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-06-29T20:34:44.972 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:01:22.905 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-06-29T09:01:22.906 INFO:teuthology.orchestra.run:waiting for 300 , 2020-06-29T09:01:22.921 INFO:tasks.ceph.mgr.x:Stopped 2020-06-29T20:34:44.972 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:16.0.0-2924.g3a96153.el8 @ceph 40 M , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:02:47.759 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-06-29T20:34:44.973 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:50.394 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.452 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/5 , 2020-06-29T09:02:50.534 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : socat-1.7.3.2-6.el8.x86_64 4/5 2020-06-29T20:34:44.974 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:16.0.0-2924.g3a96153.el8.x86_64 1/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:16.0.0-2924.g3a96153.el8.x86_64 2/4 , 2020-06-29T09:02:53.344 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:16.0.0-2924.g3a96153.el8.x86_64 3/4 2020-06-29T20:34:44.974 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch0.8.1-17.el8 @epel 24 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch3.4.0.3-18.el8 @epel 43 k , 2020-06-29T09:02:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch1.14.1-5.el8 @epel 82 k 2020-06-29T20:34:44.975 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch1.24.2-2.el8 @CentOS-Base 604 k , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch1.2.1-2.el8.1 @epel 1.0 M , 2020-06-29T09:02:54.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings noarch0.5.1-6.el8 @CentOS-AppStream 72 k 2020-06-29T20:34:44.976 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:01.258 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-portend-2.6-1.el8.noarch 26/85 , 2020-06-29T09:03:01.330 INFO:teuthology.orchestra.run.smithi176.stdout: Erasing : python3-rsa-3.4.2-12.el8.noarch 30/85 , 2020-06-29T09:03:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: Erasing : python3-tempora-1.14.1-5.el8.noarch 27/85 2020-06-29T20:34:44.976 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run , 2020-06-29T09:03:22.791 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2020-06-29T20:34:44.977 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 45/85 , 2020-06-29T09:03:26.303 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 46/85 , 2020-06-29T09:03:26.304 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 47/85 2020-06-29T20:34:44.977 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-11.0.0-2.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch , 2020-06-29T09:03:27.883 INFO:teuthology.orchestra.run.smithi176.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-29T20:34:44.978 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 61/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 62/85 , 2020-06-29T09:03:28.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:16.0.0-2924.g3a96153.el8.x86_64 63/85 2020-06-29T20:34:44.979 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 , 2020-06-29T09:03:29.765 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch , 2020-06-29T09:03:29.766 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-29T20:34:44.979 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. , 2020-06-29T09:03:32.458 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! , 2020-06-29T09:03:32.847 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-06-29T20:34:44.980 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout:Removed: , 2020-06-29T09:03:36.120 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.0.0-2924.g3a96153.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-29T20:34:44.981 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:39.036 INFO:teuthology.orchestra.run.smithi118.stdout: , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary , 2020-06-29T09:03:39.037 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-06-29T20:34:44.981 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ , 2020-06-29T09:03:42.359 INFO:teuthology.orchestra.run.smithi118.stdout:Removing: 2020-06-29T20:34:44.982 WARNING:scrape:Ignoring malparsed backtrace: No packages marked for removal. , 2020-06-29T09:03:45.975 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. , 2020-06-29T09:03:45.976 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2020-06-29T20:34:44.983 WARNING:scrape:Ignoring malparsed backtrace: 2020-06-29T09:03:50.349 INFO:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , 2020-06-29T09:03:50.356 INFO:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service 2020-06-29T20:34:45.282 INFO:scrape:Found 7 distinct failure reasons 2020-06-29T20:34:45.283 INFO:scrape:Failure: Command failed on smithi137 with status 1: 'cd /home/ubuntu/cephtest/cos && chmod +x *.sh' 2020-06-29T20:34:45.284 INFO:scrape:5 jobs: ['5189400', '5189416', '5189397', '5189398', '5189408'] 2020-06-29T20:34:45.284 INFO:scrape:suites intersection: [] 2020-06-29T20:34:45.285 INFO:scrape:suites union: ['1-install/luminous-v1only', '1-install/mimic', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{centos_7.6}', 'msgr-failures/few', 'msgr-failures/osd-delay', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'openstack', 'rados', 'rados/perf/{ceph', 'rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'settings/optimized', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'three-plus-one}', 'ubuntu_latest', 'workloads/cosbench_64K_read_write}', 'workloads/cosbench_64K_write}', 'workloads/rbd_cls}'] 2020-06-29T20:34:45.285 INFO:scrape: 2020-06-29T20:34:45.286 INFO:scrape:Dead: b'2020-06-29T20:32:22.446 DEBUG:teuthology.exit:Finished running handlers' 2020-06-29T20:34:45.286 INFO:scrape:2 jobs: ['5189409', '5189401'] 2020-06-29T20:34:45.286 INFO:scrape:suites intersection: ['2-start-upgrade', '3-wait', 'fixed-2}', 'rados/cephadm/upgrade/{1-start'] 2020-06-29T20:34:45.287 INFO:scrape:suites union: ['2-start-upgrade', '3-wait', 'distro$/{ubuntu_18.04}', 'distro$/{ubuntu_latest}', 'fixed-2}', 'rados/cephadm/upgrade/{1-start'] 2020-06-29T20:34:45.287 INFO:scrape: 2020-06-29T20:34:45.288 INFO:scrape:Failure: "2020-06-29T08:40:19.430841+0000 osd.2 (osd.2) 3 : cluster [WRN] slow request osd_op(client.4290.0:1 2.1 2.5c6d0981 (undecoded) ondisk+write+known_if_redirected e17) initiated 2020-06-29T08:39:49.235187+0000 currently delayed" in cluster log 2020-06-29T20:34:45.288 INFO:scrape:1 jobs: ['5189403'] 2020-06-29T20:34:45.288 INFO:scrape:suites: ['msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/random-eio', 'supported-random-distro$/{ubuntu_latest}}'] 2020-06-29T20:34:45.289 INFO:scrape: 2020-06-29T20:34:45.289 INFO:scrape:Failure: "2020-06-29T09:32:36.658523+0000 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi085:x (4774), after 301.381 seconds" in cluster log 2020-06-29T20:34:45.290 INFO:scrape:1 jobs: ['5189399'] 2020-06-29T20:34:45.290 INFO:scrape:suites: ['1-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'bluestore-bitmap', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}'] 2020-06-29T20:34:45.290 INFO:scrape: 2020-06-29T20:34:45.291 INFO:scrape:Failure: Command failed on smithi040 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-filestore 20 --debug-bluestore 20" ceph_test_objectstore --gtest_filter=-*/3 --gtest_catch_exceptions=0\'' 2020-06-29T20:34:45.291 INFO:scrape:1 jobs: ['5189396'] 2020-06-29T20:34:45.292 INFO:scrape:suites: ['rados/objectstore/{backends/objectstore', 'supported-random-distro$/{rhel_8}}'] 2020-06-29T20:34:45.292 INFO:scrape: 2020-06-29T20:34:45.292 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-06-29T20:34:45.293 INFO:scrape:1 jobs: ['5189402'] 2020-06-29T20:34:45.293 INFO:scrape:suites: ['1-install/nautilus', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{centos_7.6}', 'msgr-failures/osd-delay', 'rados', 'rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'thrashers/careful', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}'] 2020-06-29T20:34:45.294 INFO:scrape: 2020-06-29T20:34:45.294 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-06-29T20:34:45.294 INFO:scrape:1 jobs: ['5189410'] 2020-06-29T20:34:45.295 INFO:scrape:suites: ['msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{centos_8}}'] 2020-06-29T20:34:45.295 INFO:scrape: 2020-06-29T20:34:45.297 INFO:teuthology.results:Sending results to dupadhya@redhat.com: Test Run: ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/ failed: 10 dead: 2 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [5189396] rados/objectstore/{backends/objectstore supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 04:35:36 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189396/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189396/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=287f313f5cbb4155a42d7244b8144153 Command failed on smithi040 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to- stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug- filestore 20 --debug-bluestore 20" ceph_test_objectstore --gtest_filter=-*/3 --gtest_catch_exceptions=0\'' [5189397] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:08:59 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189397/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189397/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=38f67e4b53a744dcb6931db94b98acbe Command failed on smithi016 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [5189398] rados/perf/{ceph objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} ----------------------------------------------------------------- time: 00:06:54 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189398/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189398/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=dd0587003f9043b5bedf521f9d133246 Command failed on smithi074 with status 1: 'cd /home/ubuntu/cephtest/cos && chmod +x *.sh' [5189399] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 03:45:16 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189399/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189399/ "2020-06-29T09:32:36.658523+0000 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi085:x (4774), after 301.381 seconds" in cluster log [5189400] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/cosbench_64K_write} ----------------------------------------------------------------- time: 00:06:40 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189400/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189400/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=94d48018059846f49178e30277deb682 Command failed on smithi137 with status 1: 'cd /home/ubuntu/cephtest/cos && chmod +x *.sh' [5189402] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:25:18 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189402/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189402/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=fc1cc919c78047bc973d418e5992ccf4 reached maximum tries (180) after waiting for 180 seconds [5189403] rados/singleton/{all/random-eio msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:41:33 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189403/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189403/ "2020-06-29T08:40:19.430841+0000 osd.2 (osd.2) 3 : cluster [WRN] slow request osd_op(client.4290.0:1 2.1 2.5c6d0981 (undecoded) ondisk+write+known_if_redirected e17) initiated 2020-06-29T08:39:49.235187+0000 currently delayed" in cluster log [5189408] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:08:48 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189408/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189408/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=a79b5885eabd48d880151370353d58fd Command failed on smithi098 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' [5189410] rados/singleton/{all/thrash_cache_writeback_proxy_none msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:12:24 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189410/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189410/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5189416] rados/perf/{ceph objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} ----------------------------------------------------------------- time: 00:06:50 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189416/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189416/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=0a76163aed4144ad8c544a89da5d8964 Command failed on smithi087 with status 1: 'cd /home/ubuntu/cephtest/cos && chmod +x *.sh' Dead ================================================================= [5189401] rados/cephadm/upgrade/{1-start 2-start-upgrade 3-wait distro$/{ubuntu_latest} fixed-2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189401/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189401/ [5189409] rados/cephadm/upgrade/{1-start 2-start-upgrade 3-wait distro$/{ubuntu_18.04} fixed-2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189409/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189409/ Pass ================================================================= [5189404] rados/monthrash/{ceph clusters/9-mons msgr-failures/mon-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/pool-create-delete} time: 00:37:52 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189404/ [5189405] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/prometheus} time: 00:09:54 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189405/ [5189406] rados/singleton/{all/resolve_stuck_peering msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:07:16 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189406/ [5189407] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/valgrind} time: 00:28:42 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189407/ [5189411] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} d-balancer/on msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/dedup_tier} time: 00:11:07 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189411/ [5189412] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:38:39 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189412/ [5189413] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:18:08 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189413/ [5189414] rados/singleton-nomsgr/{all/multi-backfill-reject rados supported-random-distro$/{ubuntu_latest}} time: 00:21:40 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189414/ [5189415] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-overwrites} time: 00:20:35 info: http://pulpito.ceph.com/ideepika-2020-06-29_08:23:54-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5189415/