2020-08-26T06:24:24.505 INFO:root:teuthology version: 1.0.0-3b2e857 2020-08-26T06:24:24.508 INFO:teuthology.results:subset: '859/9999' 2020-08-26T06:24:24.508 INFO:teuthology.results:seed: '2477' 2020-08-26T06:58:27.658 INFO:root:teuthology version: 1.0.0-3b2e857 2020-08-26T06:58:27.705 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-26T10:01:32.914 INFO:teuthology.results:Tests finished! gathering results... 2020-08-26T10:02:09.784 INFO:scrape:Found 99 jobs 2020-08-26T10:03:22.055 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4755-g7c20c6a (7c20c6abc4adba80f9134e26935f00a1a3fc9711) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f7874040da9] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f7874040f84] 2020-08-26T10:03:22.060 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:03:59.630 INFO:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd deep-scrub 5 , instructed osd(s) 5 to deep-scrub , 2020-08-26T07:03:59.910 INFO:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-08-26T10:03:22.061 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:05:22.706 DEBUG:tasks.ceph.osd.3:waiting for process to exit , 2020-08-26T07:05:22.707 INFO:teuthology.orchestra.run:waiting for 300 , 2020-08-26T07:05:22.726 INFO:tasks.ceph.osd.3:Stopped 2020-08-26T10:03:22.061 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:06:42.946 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... , 2020-08-26T07:06:43.147 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... , 2020-08-26T07:06:43.147 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2020-08-26T10:03:22.062 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 70% , (Reading database ... 75% , (Reading database ... 80% 2020-08-26T10:03:22.062 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:06:48.245 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: , 2020-08-26T07:06:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon ceph-osd python3-jinja2 , 2020-08-26T07:06:48.247 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2020-08-26T10:03:22.063 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 80% , (Reading database ... 85% , (Reading database ... 90% 2020-08-26T10:03:22.064 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 95% , (Reading database ... 100% , (Reading database ... 109774 files and directories currently installed.) 2020-08-26T10:03:22.064 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:06:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-26T07:06:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-26T07:06:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-26T10:03:22.065 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:01.618 INFO:teuthology.orchestra.run.smithi105.stdout:Purging configuration files for ceph-osd (16.0.0-4755-g7c20c6a-1bionic) ... , 2020-08-26T07:07:01.738 INFO:teuthology.orchestra.run.smithi131.stdout:(Reading database ... , (Reading database ... 5% 2020-08-26T10:03:22.065 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 30% , (Reading database ... 35% , (Reading database ... 40% 2020-08-26T10:03:22.066 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:10.334 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 libsnappy1v5 libxslt1.1 python-attr python-funcsigs , 2020-08-26T07:07:10.334 INFO:teuthology.orchestra.run.smithi105.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest python3-attr , 2020-08-26T07:07:10.334 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt python3-bs4 python3-ceph-common python3-cherrypy3 2020-08-26T10:03:22.067 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:11.379 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: , 2020-08-26T07:07:11.381 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-26T07:07:11.381 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-26T10:03:22.069 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:13.670 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... , 2020-08-26T07:07:13.913 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... , 2020-08-26T07:07:13.919 INFO:teuthology.orchestra.run.smithi131.stdout:Building dependency tree... 2020-08-26T10:03:22.070 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:14.626 INFO:teuthology.orchestra.run.smithi131.stdout:0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. , W: --force-yes is deprecated, use one of the options starting with --allow instead. , 2020-08-26T07:07:14.688 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2020-08-26T10:03:22.070 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:15.602 INFO:teuthology.orchestra.run.smithi131.stdout:Removing python3-rbd (16.0.0-4755-g7c20c6a-1bionic) ... , W: --force-yes is deprecated, use one of the options starting with --allow instead. , 2020-08-26T07:07:16.416 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2020-08-26T10:03:22.071 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 107639 files and directories currently installed.) , 2020-08-26T07:07:17.656 INFO:teuthology.orchestra.run.smithi131.stdout:Removing libcephfs-dev (16.0.0-4755-g7c20c6a-1bionic) ... , 2020-08-26T07:07:17.708 INFO:teuthology.orchestra.run.smithi105.stdout:Removing libcephfs2 (16.0.0-4755-g7c20c6a-1bionic) ... 2020-08-26T10:03:22.072 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:19.986 INFO:teuthology.orchestra.run.smithi131.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu , 2020-08-26T07:07:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 , 2020-08-26T07:07:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2020-08-26T10:03:22.072 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:24.832 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-26T10:03:22.073 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:27.657 INFO:teuthology.orchestra.run.smithi131.stdout: libopus0 liborc-0.4-0 librabbitmq4 libraw1394-11 librdkafka1 libsamplerate0 , 2020-08-26T07:07:27.657 INFO:teuthology.orchestra.run.smithi131.stdout: libshout3 libsnappy1v5 libspeex1 libspice-server1 libtag1v5 , 2020-08-26T07:07:27.657 INFO:teuthology.orchestra.run.smithi131.stdout: libtag1v5-vanilla libtwolame0 libusbredirparser1 libv4l-0 libv4lconvert0 2020-08-26T10:03:22.073 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:28.363 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... , 2020-08-26T07:07:28.381 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... , 2020-08-26T07:07:28.545 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages will be REMOVED: 2020-08-26T10:03:22.074 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:29.232 INFO:teuthology.orchestra.run.smithi105.stdout:dpkg: warning: while removing ceph-mgr-modules-core, directory '/usr/share/ceph/mgr/orchestrator' not empty so not removed , 2020-08-26T07:07:29.232 INFO:teuthology.orchestra.run.smithi105.stdout:dpkg: warning: while removing ceph-mgr-modules-core, directory '/usr/share/ceph/mgr/localpool' not empty so not removed , 2020-08-26T07:07:29.233 INFO:teuthology.orchestra.run.smithi105.stdout:dpkg: warning: while removing ceph-mgr-modules-core, directory '/usr/share/ceph/mgr/iostat' not empty so not removed 2020-08-26T10:03:22.075 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:42.637 INFO:teuthology.orchestra.run.smithi131.stdout:Removing libshout3:amd64 (2.4.1-2build1) ... , 2020-08-26T07:07:42.837 INFO:teuthology.orchestra.run.smithi105.stdout:Removing libsamplerate0:amd64 (0.1.9-1) ... , 2020-08-26T07:07:42.991 INFO:teuthology.orchestra.run.smithi131.stdout:Removing libsnappy1v5:amd64 (1.1.7-1) ... 2020-08-26T10:03:22.075 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:08:05.822 INFO:teuthology.orchestra.run.smithi131.stdout:Removing python3-jwt (1.5.3+ds1-1) ... , 2020-08-26T07:08:06.141 INFO:teuthology.orchestra.run.smithi105.stdout:Removing python3-jsonpatch (1.19+really1.16-1fakesync1) ... , 2020-08-26T07:08:06.292 INFO:teuthology.orchestra.run.smithi131.stdout:Removing python3-logutils (0.3.3-5) ... 2020-08-26T10:03:22.076 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:08:27.276 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... , 2020-08-26T07:08:27.280 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... , 2020-08-26T07:08:27.323 DEBUG:teuthology.parallel:result is None 2020-08-26T10:03:22.077 WARNING:scrape:Ignoring malparsed backtrace: r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() 2020-08-26T10:03:22.294 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4755-g7c20c6a (7c20c6abc4adba80f9134e26935f00a1a3fc9711) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f7874040da9] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f7874040f84] 2020-08-26T10:03:22.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:03:59.630 INFO:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd deep-scrub 5 , instructed osd(s) 5 to deep-scrub , 2020-08-26T07:03:59.910 INFO:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-08-26T10:03:22.300 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:05:22.706 DEBUG:tasks.ceph.osd.3:waiting for process to exit , 2020-08-26T07:05:22.707 INFO:teuthology.orchestra.run:waiting for 300 , 2020-08-26T07:05:22.726 INFO:tasks.ceph.osd.3:Stopped 2020-08-26T10:03:22.301 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:06:42.946 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... , 2020-08-26T07:06:43.147 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... , 2020-08-26T07:06:43.147 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2020-08-26T10:03:22.302 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 70% , (Reading database ... 75% , (Reading database ... 80% 2020-08-26T10:03:22.303 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:06:48.245 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: , 2020-08-26T07:06:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon ceph-osd python3-jinja2 , 2020-08-26T07:06:48.247 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2020-08-26T10:03:22.304 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 80% , (Reading database ... 85% , (Reading database ... 90% 2020-08-26T10:03:22.306 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 95% , (Reading database ... 100% , (Reading database ... 109774 files and directories currently installed.) 2020-08-26T10:03:22.306 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:06:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-26T07:06:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-26T07:06:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-26T10:03:22.307 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:01.618 INFO:teuthology.orchestra.run.smithi105.stdout:Purging configuration files for ceph-osd (16.0.0-4755-g7c20c6a-1bionic) ... , 2020-08-26T07:07:01.738 INFO:teuthology.orchestra.run.smithi131.stdout:(Reading database ... , (Reading database ... 5% 2020-08-26T10:03:22.308 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 30% , (Reading database ... 35% , (Reading database ... 40% 2020-08-26T10:03:22.309 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:10.334 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 libsnappy1v5 libxslt1.1 python-attr python-funcsigs , 2020-08-26T07:07:10.334 INFO:teuthology.orchestra.run.smithi105.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest python3-attr , 2020-08-26T07:07:10.334 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt python3-bs4 python3-ceph-common python3-cherrypy3 2020-08-26T10:03:22.310 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:11.379 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: , 2020-08-26T07:07:11.381 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-26T07:07:11.381 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-26T10:03:22.311 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:13.670 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... , 2020-08-26T07:07:13.913 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... , 2020-08-26T07:07:13.919 INFO:teuthology.orchestra.run.smithi131.stdout:Building dependency tree... 2020-08-26T10:03:22.312 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:14.626 INFO:teuthology.orchestra.run.smithi131.stdout:0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. , W: --force-yes is deprecated, use one of the options starting with --allow instead. , 2020-08-26T07:07:14.688 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2020-08-26T10:03:22.314 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:15.602 INFO:teuthology.orchestra.run.smithi131.stdout:Removing python3-rbd (16.0.0-4755-g7c20c6a-1bionic) ... , W: --force-yes is deprecated, use one of the options starting with --allow instead. , 2020-08-26T07:07:16.416 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2020-08-26T10:03:22.315 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 107639 files and directories currently installed.) , 2020-08-26T07:07:17.656 INFO:teuthology.orchestra.run.smithi131.stdout:Removing libcephfs-dev (16.0.0-4755-g7c20c6a-1bionic) ... , 2020-08-26T07:07:17.708 INFO:teuthology.orchestra.run.smithi105.stdout:Removing libcephfs2 (16.0.0-4755-g7c20c6a-1bionic) ... 2020-08-26T10:03:22.316 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:19.986 INFO:teuthology.orchestra.run.smithi131.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu , 2020-08-26T07:07:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 , 2020-08-26T07:07:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2020-08-26T10:03:22.317 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:24.832 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-26T10:03:22.318 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:27.657 INFO:teuthology.orchestra.run.smithi131.stdout: libopus0 liborc-0.4-0 librabbitmq4 libraw1394-11 librdkafka1 libsamplerate0 , 2020-08-26T07:07:27.657 INFO:teuthology.orchestra.run.smithi131.stdout: libshout3 libsnappy1v5 libspeex1 libspice-server1 libtag1v5 , 2020-08-26T07:07:27.657 INFO:teuthology.orchestra.run.smithi131.stdout: libtag1v5-vanilla libtwolame0 libusbredirparser1 libv4l-0 libv4lconvert0 2020-08-26T10:03:22.319 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:28.363 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... , 2020-08-26T07:07:28.381 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... , 2020-08-26T07:07:28.545 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages will be REMOVED: 2020-08-26T10:03:22.320 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:29.232 INFO:teuthology.orchestra.run.smithi105.stdout:dpkg: warning: while removing ceph-mgr-modules-core, directory '/usr/share/ceph/mgr/orchestrator' not empty so not removed , 2020-08-26T07:07:29.232 INFO:teuthology.orchestra.run.smithi105.stdout:dpkg: warning: while removing ceph-mgr-modules-core, directory '/usr/share/ceph/mgr/localpool' not empty so not removed , 2020-08-26T07:07:29.233 INFO:teuthology.orchestra.run.smithi105.stdout:dpkg: warning: while removing ceph-mgr-modules-core, directory '/usr/share/ceph/mgr/iostat' not empty so not removed 2020-08-26T10:03:22.321 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:07:42.637 INFO:teuthology.orchestra.run.smithi131.stdout:Removing libshout3:amd64 (2.4.1-2build1) ... , 2020-08-26T07:07:42.837 INFO:teuthology.orchestra.run.smithi105.stdout:Removing libsamplerate0:amd64 (0.1.9-1) ... , 2020-08-26T07:07:42.991 INFO:teuthology.orchestra.run.smithi131.stdout:Removing libsnappy1v5:amd64 (1.1.7-1) ... 2020-08-26T10:03:22.322 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:08:05.822 INFO:teuthology.orchestra.run.smithi131.stdout:Removing python3-jwt (1.5.3+ds1-1) ... , 2020-08-26T07:08:06.141 INFO:teuthology.orchestra.run.smithi105.stdout:Removing python3-jsonpatch (1.19+really1.16-1fakesync1) ... , 2020-08-26T07:08:06.292 INFO:teuthology.orchestra.run.smithi131.stdout:Removing python3-logutils (0.3.3-5) ... 2020-08-26T10:03:22.323 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-26T07:08:27.276 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... , 2020-08-26T07:08:27.280 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... , 2020-08-26T07:08:27.323 DEBUG:teuthology.parallel:result is None 2020-08-26T10:03:22.325 WARNING:scrape:Ignoring malparsed backtrace: r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() 2020-08-26T10:03:29.196 INFO:scrape:Found 10 distinct failure reasons 2020-08-26T10:03:29.197 INFO:scrape:Failure: HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out',)) 2020-08-26T10:03:29.198 INFO:scrape:5 jobs: ['5376834', '5376827', '5376696', '5376825', '5376702'] 2020-08-26T10:03:29.199 INFO:scrape:suites intersection: ['rados'] 2020-08-26T10:03:29.199 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'fast/fast', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/fastread', 'thrashers/morepggrow', 'thrashers/none', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-small-objects-balanced}', 'workloads/pool-snaps-few-objects}'] 2020-08-26T10:03:29.200 INFO:scrape: 2020-08-26T10:03:29.200 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' 2020-08-26T10:03:29.200 INFO:scrape:48 jobs 2020-08-26T10:03:29.202 INFO:scrape:suites intersection: ['rados'] 2020-08-26T10:03:29.203 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/3-mons', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'fast/normal', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/mon-delay', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/monthrash/{ceph', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashers/sync-many', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-snaps-few-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}'] 2020-08-26T10:03:29.203 INFO:scrape: 2020-08-26T10:03:29.204 INFO:scrape:Failure: Command failed on smithi144 with status 1: 'sudo yum -y install rbd-mirror' 2020-08-26T10:03:29.204 INFO:scrape:3 jobs: ['5376821', '5376816', '5376823'] 2020-08-26T10:03:29.205 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'supported-random-distro$/{centos_8}', 'thrashosds-health'] 2020-08-26T10:03:29.206 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/careful', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/snaps-few-objects-localized}'] 2020-08-26T10:03:29.206 INFO:scrape: 2020-08-26T10:03:29.206 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2020-08-26T10:03:29.207 INFO:scrape:1 jobs: ['5376708'] 2020-08-26T10:03:29.207 INFO:scrape:suites: ['msgr-failures/many', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_8}}'] 2020-08-26T10:03:29.208 INFO:scrape: 2020-08-26T10:03:29.208 INFO:scrape:Failure: 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2020-08-26-1042 2020-08-26T10:03:29.209 INFO:scrape:12 jobs: ['5376771', '5376700', '5376769', '5376734', '5376736', '5376712', '5376747', '5376727', '5376710', '5376732', '5376701', '5376713'] 2020-08-26T10:03:29.209 INFO:scrape:suites intersection: ['rados', 'thrashosds-health'] 2020-08-26T10:03:29.209 INFO:scrape:suites union: ['1-install/luminous-v1only', '1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'clusters/{fixed-4', 'clusters/{openstack', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'distro$/{centos_7.6}', 'fast/fast', 'fast/normal', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/small-objects-balanced}'] 2020-08-26T10:03:29.210 INFO:scrape: 2020-08-26T10:03:29.210 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 0 --op append_excl 50 --pool unique_pool_0' 2020-08-26T10:03:29.211 INFO:scrape:8 jobs: ['5376692', '5376808', '5376714', '5376801', '5376703', '5376803', '5376719', '5376716'] 2020-08-26T10:03:29.211 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health'] 2020-08-26T10:03:29.212 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'fast/normal', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/minsize_recovery', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}'] 2020-08-26T10:03:29.212 INFO:scrape: 2020-08-26T10:03:29.213 INFO:scrape:Failure: {'Failure object was': {'smithi195.front.sepia.ceph.com': {'msg': 'Failed to download packages: Cannot download Packages/boost-random-1.66.0-6.el8.x86_64.rpm: All mirrors were tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu-kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"} 2020-08-26T10:03:29.213 INFO:scrape:6 jobs: ['5376845', '5376847', '5376851', '5376829', '5376838', '5376836'] 2020-08-26T10:03:29.214 INFO:scrape:suites intersection: [] 2020-08-26T10:03:29.214 INFO:scrape:suites union: ['1-install/nautilus', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-recovery-overrides/{default}', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'backoff/normal', 'backoff/peering_and_degraded', 'bluestore-bitmap', 'ceph', 'cluster/{12-osds', 'clusters/3-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/normal', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/mon-delay', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/monthrash/{ceph', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'recovery-overrides/{default}', 'recovery-overrides/{more-async-recovery}', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashers/sync', 'thrashosds-health', 'ubuntu_latest}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-small-objects-fast-read}', 'workloads/snaps-few-objects}'] 2020-08-26T10:03:29.214 INFO:scrape: 2020-08-26T10:03:29.215 INFO:scrape:Failure: Command failed (workunit test mon/mon-handle-forward.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-handle-forward.sh' 2020-08-26T10:03:29.215 INFO:scrape:1 jobs: ['5376738'] 2020-08-26T10:03:29.216 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] 2020-08-26T10:03:29.216 INFO:scrape: 2020-08-26T10:03:29.217 INFO:scrape:Failure: Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' 2020-08-26T10:03:29.217 INFO:scrape:5 jobs: ['5376840', '5376688', '5376791', '5376693', '5376743'] 2020-08-26T10:03:29.217 INFO:scrape:suites intersection: ['rados'] 2020-08-26T10:03:29.218 INFO:scrape:suites union: ['msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] 2020-08-26T10:03:29.218 INFO:scrape: 2020-08-26T10:03:29.219 INFO:scrape:Failure: Command failed (workunit test osd/osd-rep-recov-eio.sh) on smithi029 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh' 2020-08-26T10:03:29.219 INFO:scrape:1 jobs: ['5376782'] 2020-08-26T10:03:29.219 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_8}', 'workloads/osd}'] 2020-08-26T10:03:29.220 INFO:scrape: 2020-08-26T10:03:29.228 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/ failed: 84 dead: 6 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [5376685] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:14:18 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376685/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376685/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376687] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/fastclose rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:09:14 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376687/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376687/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376688] rados/singleton-bluestore/{all/cephtool msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:28:29 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376688/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376688/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=30e24af7cf864f74be56afa403f14fee Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi001 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5376690] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:09:07 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376690/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376690/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376691] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:13:46 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376691/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376691/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376692] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-small-objects-balanced} ----------------------------------------------------------------- time: 00:08:30 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376692/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376692/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 0 --op append_excl 50 --pool unique_pool_0' [5376693] rados/singleton-nomsgr/{all/ceph-kvstore-tool rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:07:47 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376693/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376693/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=d3de0b5b6d08453fa6c2658f3fda3f4e Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5376694] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:11:29 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376694/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376694/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376695] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:08:55 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376695/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376695/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376696] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:05:40 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376696/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376696/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=2cc5047bcd6543518b0e6e42800bfde6 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86 _64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 [5376697] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:11:00 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376697/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376697/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376698] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:13:16 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376698/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376698/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376699] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:12:39 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376699/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376699/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376700] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:04:02 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376700/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376700/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=9a79a6e0a2584c0292220b0dbe1f5c90 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2 Fx86_64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 [5376701] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:02:39 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376701/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376701/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=7dcb6b53a17047a7ac0f27e9c09545e4 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2 Fx86_64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 [5376702] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:02:27 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376702/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376702/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=f334e24fd3d744928c3183fdf80f2493 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86 _64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 [5376703] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read} ----------------------------------------------------------------- time: 00:12:04 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376703/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376703/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 0 --op append_excl 50 --pool unique_pool_0' [5376704] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:12:18 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376704/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376704/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376705] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:09:24 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376705/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376705/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376707] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:10:01 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376707/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376707/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376708] rados/singleton-bluestore/{all/cephtool msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:18:12 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376708/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376708/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=b7ec1c274c444e6db71237d54b78b4ba Command failed (workunit test cephtool/test.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [5376709] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:08:31 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376709/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376709/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376710] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:02:07 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376710/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376710/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=a4d9785ef46c490697ff98707a7b132f 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2 Fx86_64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 [5376711] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:08:45 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376711/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376711/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376712] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:02:08 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376712/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376712/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=0a60a4a2bd0a4f6ca99fd01d2adf7259 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2 Fx86_64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 [5376713] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:02:39 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376713/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376713/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=53c50d0183c2408d9c7d9f6aba7c09d7 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2 Fx86_64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 [5376714] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:10:12 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376714/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376714/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376715] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:13:48 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376715/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376715/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376716] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:13:17 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376716/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376716/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376717] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:09:12 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376717/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376717/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376719] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:15:04 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376719/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376719/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 0 --op append_excl 50 --pool unique_pool_0' [5376721] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:08:44 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376721/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376721/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376723] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:10:27 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376723/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376723/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376725] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:08:54 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376725/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376725/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376727] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:05:55 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376727/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376727/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=2bb459b12f89483a955bce742de4d7f9 500 Server Error: Internal Server Error for url: https://4.chacra.ceph.com/ repos/ceph/luminous/5a69e6bc606a4975a2e558e52ca47939037e98bb/centos/7/flavo rs/default/repo [5376730] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:08:56 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376730/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376730/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376732] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376732/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376732/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=b8a3bd1b0dbb4d5e8cfb9566dfed5685 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2 Fx86_64&ref=wip-kefu-testing-2020-08-26-1042 [5376734] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376734/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376734/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=b431ae36cd5a4243a78be039b16797a3 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86 _64&ref=wip-kefu-testing-2020-08-26-1042 [5376736] rados/cephadm/upgrade/{1-start 2-start-upgrade 3-wait distro$/{ubuntu_latest} fixed-2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376736/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376736/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=2717065855fa474783c378681534110b 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2 Fx86_64&ref=wip-kefu-testing-2020-08-26-1042 [5376738] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} ----------------------------------------------------------------- time: 00:11:27 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376738/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376738/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=15621b4781c044c6a8bdab13c4d51e61 Command failed (workunit test mon/mon-handle-forward.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-handle- forward.sh' [5376740] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:13:01 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376740/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376740/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376743] rados/singleton-bluestore/{all/cephtool msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:32:39 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376743/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376743/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=34243018a042422faa45b6a730c58a10 Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5376745] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:08:31 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376745/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376745/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376747] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376747/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376747/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=b4ac41bd4a1042c4a3a921d17738996a 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86 _64&ref=wip-kefu-testing-2020-08-26-1042 [5376749] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:15:20 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376749/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376749/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376754] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:11:05 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376754/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376754/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376756] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:10:01 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376756/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376756/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376758] rados/singleton/{all/thrash_cache_writeback_proxy_none msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:18:14 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376758/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376758/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5376760] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:12:54 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376760/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376760/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376764] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:14:20 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376764/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376764/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376767] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:13:59 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376767/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376767/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376769] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376769/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376769/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=44292530ec0c4721bae1f663f10de96a 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86 _64&ref=wip-kefu-testing-2020-08-26-1042 [5376771] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376771/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376771/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=fd4d9a336cf347a780e89e60f0393df6 500 Server Error: Internal Server Error for url: https://shaman.ceph.com/ap i/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86 _64&ref=wip-kefu-testing-2020-08-26-1042 [5376773] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:12:25 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376773/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376773/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376775] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:09:04 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376775/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376775/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376777] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:12:48 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376777/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376777/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376780] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:14:41 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376780/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376780/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376782] rados/standalone/{supported-random-distro$/{centos_8} workloads/osd} ----------------------------------------------------------------- time: 03:09:43 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376782/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376782/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=9e1dcec2f1714bacae32864a3b9f5ace Command failed (workunit test osd/osd-rep-recov-eio.sh) on smithi029 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov- eio.sh' [5376784] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} ----------------------------------------------------------------- time: 00:08:29 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376784/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376784/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376786] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:14:41 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376786/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376786/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376788] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:20:09 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376788/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376788/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376791] rados/singleton-bluestore/{all/cephtool msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:32:34 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376791/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376791/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=63b1e42c87b2449f91efe04031718bfb Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5376793] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:08:36 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376793/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376793/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376795] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:09:56 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376795/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376795/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376801] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:19:41 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376801/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376801/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376803] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:10:05 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376803/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376803/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376806] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:07:51 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376806/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376806/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376808] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:11:51 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376808/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376808/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376810] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:08:09 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376810/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376810/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376812] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:10:12 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376812/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376812/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376814] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:10:34 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376814/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376814/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376816] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:06:35 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376816/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376816/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=cec17b2629934770967a6ec7f9f5fe6e Command failed on smithi201 with status 1: 'sudo yum -y install rbd-mirror' [5376819] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:17:13 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376819/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376819/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5376821] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:06:50 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376821/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376821/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=804c4968af074fa5bf8577acb293062f Command failed on smithi144 with status 1: 'sudo yum -y install rbd-mirror' [5376823] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:06:01 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376823/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376823/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=5f7c87ed14354cfaa06d36982a13fa9f Command failed on smithi055 with status 1: 'sudo yum -y install ceph- immutable-object-cache' [5376825] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced} ----------------------------------------------------------------- time: 00:06:08 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376825/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376825/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=8555f7e98bfb4a50838f590d157b535a HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F18.04%2Fx86_64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out',)) [5376827] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:04:16 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376827/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376827/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=4598706c45db4fce8ad1cf169864f67f HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun tu%2F18.04%2Fx86_64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out',)) [5376831] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:15:33 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376831/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376831/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376834] rados/singleton-nomsgr/{all/admin_socket_output rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:07:30 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376834/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376834/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=20ba40b3469345baae1c7f0bca13154c HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent os%2F8%2Fx86_64&sha1=7c20c6abc4adba80f9134e26935f00a1a3fc9711 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out',)) [5376840] rados/singleton-bluestore/{all/cephtool msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:35:41 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376840/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376840/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=2a003ddfa94d410aa8f9b4f605ad0de9 Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c20c6abc4adba80f9134e26935f00a1a3fc9711 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5376850] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:11:30 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376850/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376850/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376852] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:13:09 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376852/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376852/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5376853] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:11:30 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376853/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376853/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5376854] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:13:01 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376854/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376854/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' Dead ================================================================= [5376829] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:06:10 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376829/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376829/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=3e32f6134de7496ab3dacdbddb99995a {'Failure object was': {'smithi063.front.sepia.ceph.com': {'msg': 'Failed to download packages: Cannot download Packages/boost- random-1.66.0-6.el8.x86_64.rpm: All mirrors were tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu- kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te uthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"} [5376836] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:01:33 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376836/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376836/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=e38f9f9c28a3407cbfec3536094c7de5 {'Failure object was': {'smithi171.front.sepia.ceph.com': {'results': [{'msg': 'Failed to download key at http://download.ceph.com/keys/autobuild.asc: Request failed: ', 'failed': True, 'invocation': {'module_args': {'file': 'None', 'keyserver': 'None', 'url': 'http://download.ceph.com/keys/autobuild.asc', 'validate_certs': True, 'keyring': 'None', 'state': 'present', 'key': 'None', 'data': 'None', 'id': 'None'}}, '_ansible_no_log': False, 'changed': False, 'item': 'http://download.ceph.com/keys/autobuild.asc', 'ansible_loop_var': 'item', '_ansible_item_label': 'http://download.ceph.com/keys/autobuild.asc'}, {'msg': 'Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: ', 'failed': True, 'invocation': {'module_args': {'file': 'None', 'keyserver': 'None', 'url': 'http://download.ceph.com/keys/release.asc', 'validate_certs': True, 'keyring': 'None', 'state': 'present', 'key': 'None', 'data': 'None', 'id': 'None'}}, '_ansible_no_log': False, 'changed': False, 'item': 'http://download.ceph.com/keys/release.asc', 'ansible_loop_var': 'item', '_ansible_item_label': 'http://download.ceph.com/keys/release.asc'}], 'msg': 'All items completed', 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te uthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 199, in represent_list return self.represent_sequence(\'tag:yaml.org,2002:seq\', data) File "/home/teuthw orker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 92, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teut hology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'http://download.ceph.com/keys/autobuild.asc')"} [5376838] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:04:26 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376838/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376838/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=87decae5fa3c4c30b459c6681164ea3e {'Failure object was': {'smithi007.front.sepia.ceph.com': {'msg': 'Failed to download packages: Cannot download Packages/boost- random-1.66.0-6.el8.x86_64.rpm: All mirrors were tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu- kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te uthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"} [5376845] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/sync workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:04:24 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376845/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376845/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=fd1a6d8c315a4fcb915fc407e486e562 {'Failure object was': {'smithi195.front.sepia.ceph.com': {'msg': 'Failed to download packages: Cannot download Packages/boost- random-1.66.0-6.el8.x86_64.rpm: All mirrors were tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu- kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te uthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"} [5376847] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:01:24 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376847/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376847/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=79b29e215ecf4812bdd724c280d54502 {'Failure object was': {'smithi191.front.sepia.ceph.com': {'results': [{'msg': 'Failed to download key at http://download.ceph.com/keys/autobuild.asc: Request failed: ', 'failed': True, 'invocation': {'module_args': {'file': 'None', 'keyserver': 'None', 'url': 'http://download.ceph.com/keys/autobuild.asc', 'validate_certs': True, 'keyring': 'None', 'state': 'present', 'key': 'None', 'data': 'None', 'id': 'None'}}, '_ansible_no_log': False, 'changed': False, 'item': 'http://download.ceph.com/keys/autobuild.asc', 'ansible_loop_var': 'item', '_ansible_item_label': 'http://download.ceph.com/keys/autobuild.asc'}, {'msg': 'Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: ', 'failed': True, 'invocation': {'module_args': {'file': 'None', 'keyserver': 'None', 'url': 'http://download.ceph.com/keys/release.asc', 'validate_certs': True, 'keyring': 'None', 'state': 'present', 'key': 'None', 'data': 'None', 'id': 'None'}}, '_ansible_no_log': False, 'changed': False, 'item': 'http://download.ceph.com/keys/release.asc', 'ansible_loop_var': 'item', '_ansible_item_label': 'http://download.ceph.com/keys/release.asc'}], 'msg': 'All items completed', 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te uthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 199, in represent_list return self.represent_sequence(\'tag:yaml.org,2002:seq\', data) File "/home/teuthw orker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 92, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teut hology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'http://download.ceph.com/keys/autobuild.asc')"} [5376851] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read} ----------------------------------------------------------------- time: 00:01:07 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376851/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376851/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=59cbc3971b0445e484da860165db61a9 {'Failure object was': {'smithi062.front.sepia.ceph.com': {'results': [{'msg': 'Failed to download key at http://download.ceph.com/keys/autobuild.asc: Request failed: ', 'failed': True, 'invocation': {'module_args': {'file': 'None', 'keyserver': 'None', 'url': 'http://download.ceph.com/keys/autobuild.asc', 'validate_certs': True, 'keyring': 'None', 'state': 'present', 'key': 'None', 'data': 'None', 'id': 'None'}}, '_ansible_no_log': False, 'changed': False, 'item': 'http://download.ceph.com/keys/autobuild.asc', 'ansible_loop_var': 'item', '_ansible_item_label': 'http://download.ceph.com/keys/autobuild.asc'}, {'invocation': {'module_args': {'file': 'None', 'keyserver': 'None', 'url': 'http://download.ceph.com/keys/release.asc', 'validate_certs': True, 'keyring': 'None', 'state': 'present', 'key': 'None', 'data': 'None', 'id': 'None'}}, 'changed': False, '_ansible_no_log': False, 'failed': False, 'item': 'http://download.ceph.com/keys/release.asc', 'ansible_loop_var': 'item', '_ansible_item_label': 'http://download.ceph.com/keys/release.asc'}], 'msg': 'All items completed', 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te uthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 199, in represent_list return self.represent_sequence(\'tag:yaml.org,2002:seq\', data) File "/home/teuthw orker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 92, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teut hology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'http://download.ceph.com/keys/autobuild.asc')"} Pass ================================================================= [5376686] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_cls_all validater/valgrind} time: 03:11:23 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376686/ [5376689] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:09:59 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376689/ [5376706] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/valgrind} time: 01:03:44 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376706/ [5376751] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:12:03 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376751/ [5376762] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:09:32 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376762/ [5376797] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:42:00 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376797/ [5376799] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:15:51 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376799/ [5376842] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:31:04 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376842/ [5376849] rados/cephadm/upgrade/{1-start 2-start-upgrade 3-wait distro$/{centos_8.0} fixed-2} time: 00:17:51 info: http://pulpito.ceph.com/kchai-2020-08-26_05:51:42-rados-wip-kefu-testing-2020-08-26-1042-distro-basic-smithi/5376849/