2020-08-21T13:07:36.003 INFO:root:teuthology version: 1.0.0-c0ecc52 2020-08-21T13:07:36.005 INFO:teuthology.results:subset: '8561/9999' 2020-08-21T13:07:36.006 INFO:teuthology.results:seed: '2477' 2020-08-21T16:28:37.317 INFO:root:teuthology version: 1.0.0-c0ecc52 2020-08-21T16:28:37.387 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-22T03:59:05.733 INFO:teuthology.results:Tests finished! gathering results... 2020-08-22T04:10:18.102 INFO:scrape:Found 337 jobs 2020-08-22T04:11:08.985 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] 2020-08-22T04:11:08.991 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-08-22T04:11:08.992 WARNING:scrape:Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() 2020-08-22T04:11:08.992 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-08-22T04:11:08.993 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. 2020-08-22T04:11:08.994 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% 2020-08-22T04:11:08.995 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:08.995 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-22T04:11:08.996 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-22T04:11:08.997 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:08.997 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-22T04:11:08.998 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% 2020-08-22T04:11:08.999 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-08-22T04:11:09.000 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% 2020-08-22T04:11:09.000 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-22T04:11:09.001 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.002 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% 2020-08-22T04:11:09.002 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.003 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.004 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios 2020-08-22T04:11:09.004 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.005 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% 2020-08-22T04:11:09.005 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2020-08-22T04:11:09.006 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... 2020-08-22T04:11:09.007 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is 2020-08-22T04:11:09.007 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-08-22T04:11:09.161 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] 2020-08-22T04:11:09.164 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-08-22T04:11:09.165 WARNING:scrape:Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() 2020-08-22T04:11:09.166 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-08-22T04:11:09.166 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. 2020-08-22T04:11:09.167 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% 2020-08-22T04:11:09.168 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.168 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-22T04:11:09.169 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-22T04:11:09.169 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.170 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-22T04:11:09.171 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% 2020-08-22T04:11:09.171 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-08-22T04:11:09.172 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% 2020-08-22T04:11:09.172 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-22T04:11:09.173 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.174 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% 2020-08-22T04:11:09.174 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.175 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.176 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios 2020-08-22T04:11:09.177 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.177 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% 2020-08-22T04:11:09.178 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2020-08-22T04:11:09.178 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... 2020-08-22T04:11:09.179 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is 2020-08-22T04:11:09.180 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-08-22T04:11:09.342 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] 2020-08-22T04:11:09.345 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-08-22T04:11:09.346 WARNING:scrape:Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() 2020-08-22T04:11:09.347 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-08-22T04:11:09.347 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. 2020-08-22T04:11:09.348 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% 2020-08-22T04:11:09.348 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.349 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-22T04:11:09.350 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-22T04:11:09.351 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.352 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-22T04:11:09.353 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% 2020-08-22T04:11:09.354 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-08-22T04:11:09.354 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% 2020-08-22T04:11:09.355 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-22T04:11:09.356 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.357 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% 2020-08-22T04:11:09.357 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.358 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.358 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios 2020-08-22T04:11:09.359 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.360 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% 2020-08-22T04:11:09.360 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2020-08-22T04:11:09.361 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... 2020-08-22T04:11:09.362 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is 2020-08-22T04:11:09.362 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-08-22T04:11:09.519 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] 2020-08-22T04:11:09.522 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-08-22T04:11:09.523 WARNING:scrape:Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() 2020-08-22T04:11:09.524 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-08-22T04:11:09.526 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. 2020-08-22T04:11:09.527 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% 2020-08-22T04:11:09.528 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.528 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-22T04:11:09.529 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-22T04:11:09.530 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.530 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-22T04:11:09.531 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% 2020-08-22T04:11:09.532 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-08-22T04:11:09.532 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% 2020-08-22T04:11:09.533 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-22T04:11:09.533 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.534 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% 2020-08-22T04:11:09.535 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.535 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.536 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios 2020-08-22T04:11:09.537 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.538 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% 2020-08-22T04:11:09.538 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2020-08-22T04:11:09.539 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... 2020-08-22T04:11:09.539 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is 2020-08-22T04:11:09.540 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-08-22T04:11:09.699 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] 2020-08-22T04:11:09.703 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-08-22T04:11:09.704 WARNING:scrape:Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() 2020-08-22T04:11:09.704 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-08-22T04:11:09.705 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. 2020-08-22T04:11:09.705 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% 2020-08-22T04:11:09.706 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.707 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-22T04:11:09.707 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-22T04:11:09.708 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.708 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-22T04:11:09.709 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% 2020-08-22T04:11:09.710 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-08-22T04:11:09.710 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% 2020-08-22T04:11:09.711 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-22T04:11:09.712 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.712 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% 2020-08-22T04:11:09.713 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.713 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.714 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios 2020-08-22T04:11:09.715 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.715 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% 2020-08-22T04:11:09.716 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2020-08-22T04:11:09.717 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... 2020-08-22T04:11:09.717 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is 2020-08-22T04:11:09.718 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-08-22T04:11:09.867 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] 2020-08-22T04:11:09.871 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-08-22T04:11:09.872 WARNING:scrape:Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() 2020-08-22T04:11:09.872 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-08-22T04:11:09.873 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. 2020-08-22T04:11:09.874 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% 2020-08-22T04:11:09.874 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.875 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-22T04:11:09.875 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-22T04:11:09.876 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.877 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-22T04:11:09.877 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% 2020-08-22T04:11:09.878 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-08-22T04:11:09.879 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% 2020-08-22T04:11:09.879 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-22T04:11:09.880 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:09.881 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% 2020-08-22T04:11:09.881 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.882 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:09.883 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios 2020-08-22T04:11:09.883 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:09.884 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% 2020-08-22T04:11:09.885 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2020-08-22T04:11:09.885 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... 2020-08-22T04:11:09.886 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is 2020-08-22T04:11:09.887 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-08-22T04:11:10.019 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] 2020-08-22T04:11:10.022 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-08-22T04:11:10.023 WARNING:scrape:Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() 2020-08-22T04:11:10.024 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-08-22T04:11:10.024 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. 2020-08-22T04:11:10.025 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% 2020-08-22T04:11:10.026 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:10.026 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-22T04:11:10.027 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-22T04:11:10.027 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:10.028 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-22T04:11:10.029 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% 2020-08-22T04:11:10.029 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-08-22T04:11:10.030 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% 2020-08-22T04:11:10.031 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-22T04:11:10.031 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:10.032 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% 2020-08-22T04:11:10.032 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:10.033 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:10.034 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios 2020-08-22T04:11:10.034 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:10.035 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% 2020-08-22T04:11:10.036 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2020-08-22T04:11:10.036 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... 2020-08-22T04:11:10.037 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is 2020-08-22T04:11:10.037 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-08-22T04:11:10.179 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] 2020-08-22T04:11:10.182 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-08-22T04:11:10.183 WARNING:scrape:Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() 2020-08-22T04:11:10.184 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-08-22T04:11:10.184 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. 2020-08-22T04:11:10.185 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% 2020-08-22T04:11:10.185 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:10.186 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% 2020-08-22T04:11:10.187 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-22T04:11:10.187 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:10.188 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-22T04:11:10.189 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% 2020-08-22T04:11:10.189 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-08-22T04:11:10.190 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% 2020-08-22T04:11:10.190 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-22T04:11:10.191 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% 2020-08-22T04:11:10.192 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% 2020-08-22T04:11:10.192 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:10.193 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... 2020-08-22T04:11:10.194 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios 2020-08-22T04:11:10.194 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-08-22T04:11:10.195 WARNING:scrape:Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% 2020-08-22T04:11:10.195 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2020-08-22T04:11:10.196 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... 2020-08-22T04:11:10.197 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is 2020-08-22T04:11:10.197 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-08-22T04:13:15.282 INFO:scrape:Found 10 distinct failure reasons 2020-08-22T04:13:15.288 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-08-22T04:13:15.289 INFO:scrape:4 jobs: ['5365482', '5365231', '5365409', '5365307'] 2020-08-22T04:13:15.290 INFO:scrape:suites intersection: ['ceph', 'clusters/{openstack', 'distro$/{centos_7.6}', 'rados', 'thrashosds-health', 'three-plus-one}'] 2020-08-22T04:13:15.291 INFO:scrape:suites union: ['1-install/luminous', '1-install/luminous-v1only', '1-install/mimic', '1-install/nautilus', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{centos_7.6}', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}', 'workloads/test_rbd_api}'] 2020-08-22T04:13:15.291 INFO:scrape: 2020-08-22T04:13:15.292 INFO:scrape:Dead: b'2020-08-22T02:02:18.058 DEBUG:teuthology.exit:Finished running handlers' 2020-08-22T04:13:15.292 INFO:scrape:5 jobs: ['5365321', '5365215', '5365325', '5365360', '5365495'] 2020-08-22T04:13:15.293 INFO:scrape:suites intersection: [] 2020-08-22T04:13:15.294 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '2-start-upgrade', '3-wait', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'debug/mgr', 'distro$/{centos_8.0}', 'distro$/{ubuntu_latest}', 'fixed-2}', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/cephadm/upgrade/{1-start', 'rados/dashboard/{clusters/{2-node-mgr}', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/dashboard}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/set-chunk-promote-flush}'] 2020-08-22T04:13:15.294 INFO:scrape: 2020-08-22T04:13:15.295 INFO:scrape:Failure: "2020-08-21T14:43:38.297138+0000 osd.2 (osd.2) 40 : cluster [ERR] 2.8 shard 2 2:10fa4044:::benchmark_data_smithi149_15659_object110629:head : missing" in cluster log 2020-08-22T04:13:15.295 INFO:scrape:5 jobs: ['5365376', '5365412', '5365274', '5365483', '5365335'] 2020-08-22T04:13:15.296 INFO:scrape:suites intersection: ['rados'] 2020-08-22T04:13:15.297 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'fast/normal', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/multi-backfill-reject', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-agent-big}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/radosbench}'] 2020-08-22T04:13:15.297 INFO:scrape: 2020-08-22T04:13:15.298 INFO:scrape:Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi159 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' 2020-08-22T04:13:15.299 INFO:scrape:ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) 1: (()+0x12dc0) [0x7f1b5eef7dc0] 2: (BlueFS::_allocate(unsigned char, unsigned long, bluefs_fnode_t*)+0x873) [0x55c90625b463] 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x13e) [0x55c90625b73e] 4: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x110) [0x55c90625d430] 5: (BlueFS::_fsync(BlueFS::FileWriter*, std::unique_lock&)+0x159) [0x55c90625fe79] 6: (BlueRocksWritableFile::Sync()+0x6c) [0x55c906289d2c] 7: (rocksdb::WriteStringToFile(rocksdb::Env*, rocksdb::Slice const&, std::__cxx11::basic_string, std::allocator > const&, bool)+0x107) [0x55c9064681f7] 8: (rocksdb::SetIdentityFile(rocksdb::Env*, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&)+0x114) [0x55c906473244] 9: (rocksdb::DBImpl::NewDB()+0x172) [0x55c9063a3162] 10: (()+0x5ad286) [0x55c90641a286] 11: (rocksdb::RepairDB(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Options const&)+0x13d) [0x55c90641ca7d] 12: (RocksDBStore::repair(std::ostream&)+0x43f) [0x55c90631e85f] 13: (main()+0x2a6) [0x55c90606ae86] 14: (__libc_start_main()+0xf3) [0x7f1b5d59a873] 15: (_start()+0x2e) [0x55c906094a1e] 2020-08-22T04:13:15.299 INFO:scrape:6 jobs: ['5365373', '5365448', '5365298', '5365525', '5365537', '5365223'] 2020-08-22T04:13:15.300 INFO:scrape:suites intersection: ['rados'] 2020-08-22T04:13:15.301 INFO:scrape:suites union: ['msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] 2020-08-22T04:13:15.301 INFO:scrape: 2020-08-22T04:13:15.302 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi065 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2020-08-22T04:13:15.302 INFO:scrape:16 jobs: ['5365323', '5365531', '5365509', '5365438', '5365260', '5365296', '5365430', '5365408', '5365308', '5365469', '5365528', '5365355', '5365220', '5365468', '5365377', '5365328'] 2020-08-22T04:13:15.303 INFO:scrape:suites intersection: [] 2020-08-22T04:13:15.304 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/9-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'd-thrash/none', 'fixed-2', 'mode/packaged', 'mode/root', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/cephadm/with-work/{distro/centos_8.0', 'rados/cephadm/with-work/{distro/centos_latest', 'rados/cephadm/with-work/{distro/rhel_8.0', 'rados/cephadm/with-work/{distro/rhel_latest', 'rados/cephadm/with-work/{distro/ubuntu_18.04', 'rados/monthrash/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'start', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests', 'tasks/rados_api_tests}', 'thrashers/careful', 'thrashers/default', 'thrashers/many', 'thrashers/one', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}', 'workloads/rados_api_tests}'] 2020-08-22T04:13:15.304 INFO:scrape: 2020-08-22T04:13:15.305 INFO:scrape:Failure: Command failed on smithi107 with status 1: 'sudo yum -y install ceph-test' 2020-08-22T04:13:15.306 INFO:scrape:2 jobs: ['5365383', '5365259'] 2020-08-22T04:13:15.306 INFO:scrape:suites intersection: ['rados', 'supported-random-distro$/{centos_8}'] 2020-08-22T04:13:15.307 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/multimon/{clusters/3', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'tasks/mon_clock_no_skews}', 'thrashers/none', 'thrashosds-health', 'workloads/small-objects}'] 2020-08-22T04:13:15.307 INFO:scrape: 2020-08-22T04:13:15.308 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-08-22T04:13:15.308 INFO:scrape:1 jobs: ['5365510'] 2020-08-22T04:13:15.309 INFO:scrape:suites: ['msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}'] 2020-08-22T04:13:15.310 INFO:scrape: 2020-08-22T04:13:15.310 INFO:scrape:Failure: wait_for_clean: failed before timeout expired 2020-08-22T04:13:15.311 INFO:scrape:1 jobs: ['5365476'] 2020-08-22T04:13:15.311 INFO:scrape:suites: ['1-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'bluestore-bitmap', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}'] 2020-08-22T04:13:15.312 INFO:scrape: 2020-08-22T04:13:15.312 INFO:scrape:Failure: Command failed on smithi122 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-filestore 20 --debug-bluestore 20" ceph_test_objectstore --gtest_filter=-*/3 --gtest_catch_exceptions=0\'' 2020-08-22T04:13:15.313 INFO:scrape:1 jobs: ['5365437'] 2020-08-22T04:13:15.313 INFO:scrape:suites: ['rados/objectstore/{backends/objectstore', 'supported-random-distro$/{centos_8}}'] 2020-08-22T04:13:15.314 INFO:scrape: 2020-08-22T04:13:15.315 INFO:scrape:Timeout 120 running ceph --cluster ceph --log-early osd last-stat-seq osd.2 2020-08-22T04:13:15.315 INFO:scrape:1 jobs: ['5365299'] 2020-08-22T04:13:15.316 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-pool-snaps-few-objects-overwrites}'] 2020-08-22T04:13:15.316 INFO:scrape: 2020-08-22T04:13:15.356 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/ failed: 37 dead: 5 running: 0 waiting: 0 queued: 0 passed: 295 Fail ================================================================= [5365220] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:32:13 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365220/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365220/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=f2628273500b4fbc9666b090e18e4375 Command failed (workunit test rados/test.sh) on smithi069 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365223] rados/singleton-bluestore/{all/cephtool msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:29:30 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365223/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365223/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=d3555df12ac748d79759e0a77944ca6f Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi005 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5365231] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:24:57 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365231/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365231/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=2cc3d0d56be144b7a0acb59c34328f76 reached maximum tries (180) after waiting for 180 seconds [5365259] rados/multimon/{clusters/3 msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} ----------------------------------------------------------------- time: 00:04:19 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365259/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365259/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=ff1d975c65c540e399544f7f28c2f8f8 Command failed on smithi093 with status 1: 'sudo yum -y install ceph-test' [5365260] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:30:24 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365260/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365260/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=a8991e5943294e13ae8c392876f10858 Command failed (workunit test rados/test.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365274] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:32:55 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365274/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365274/ "2020-08-21T13:59:15.721070+0000 osd.1 (osd.1) 51 : cluster [ERR] 4.0 shard 6 4:2336eaf7:::smithi00235360-6484:head : missing" in cluster log [5365296] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:06:40 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365296/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365296/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=a63abf7962eb49f296139b1dcc6b9e31 Command failed (workunit test rados/test.sh) on smithi106 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365298] rados/singleton-bluestore/{all/cephtool msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:29:07 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365298/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365298/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=7a76c2d34a4f41e4885867a047a2cd52 Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi082 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5365299] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:33:06 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365299/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365299/ Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.2' [5365307] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:28:41 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365307/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365307/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=de9adb07d0da43749420b0a054251367 reached maximum tries (180) after waiting for 180 seconds [5365308] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:28:10 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365308/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365308/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=a4c63d6e875b4b98a439d992424557cc Command failed (workunit test rados/test.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365323] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:23:55 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365323/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365323/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=2664c0bc10524378844826bbc19331c6 Command failed (workunit test rados/test.sh) on smithi065 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365328] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:21:40 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365328/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365328/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=b9873d1e37354faea3a9187c63679470 Command failed (workunit test rados/test.sh) on smithi085 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365335] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} ----------------------------------------------------------------- time: 00:39:14 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365335/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365335/ "2020-08-21T14:34:27.364001+0000 osd.4 (osd.4) 230 : cluster [ERR] 3.24s1 shard 2(2) 3:27b7b36c:::smithi13934080-11:166 : missing" in cluster log [5365355] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:33:24 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365355/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365355/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=332dd7616e784666a2f7372efd7d7ca3 Command failed (workunit test rados/test.sh) on smithi035 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365373] rados/singleton-bluestore/{all/cephtool msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:28:38 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365373/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365373/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=37111b0a6c7b48ff8d4e9eace2889e5f Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi159 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5365376] rados/singleton-nomsgr/{all/multi-backfill-reject rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:21:26 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365376/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365376/ "2020-08-21T14:43:38.297138+0000 osd.2 (osd.2) 40 : cluster [ERR] 2.8 shard 2 2:10fa4044:::benchmark_data_smithi149_15659_object110629:head : missing" in cluster log [5365377] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:31:14 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365377/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365377/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=e88747d9b5f2433fa8d62f7a03cdf8db Command failed (workunit test rados/test.sh) on smithi058 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365383] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:04:22 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365383/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365383/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=45a58732da9444de8e6ab16481dff2cf Command failed on smithi107 with status 1: 'sudo yum -y install ceph-test' [5365408] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:30:26 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365408/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365408/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=f195201fe0d747bfb3758cc5f89223ab Command failed (workunit test rados/test.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365409] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:26:01 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365409/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365409/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=0061266c463e4970a70c10cc55b94f60 reached maximum tries (180) after waiting for 180 seconds [5365412] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:40:30 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365412/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365412/ "2020-08-21T15:11:21.062652+0000 osd.5 (osd.5) 16 : cluster [ERR] 3.17 shard 3 3:e997cf6d:::benchmark_data_smithi187_33766_object19058:head : missing" in cluster log [5365430] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:11:29 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365430/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365430/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=aad7a59a50fd46c283ec9a7b3fd78325 Command failed (workunit test rados/test.sh) on smithi193 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365437] rados/objectstore/{backends/objectstore supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 05:09:04 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365437/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365437/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=5f211898d4ed4bd4b5e1f1f78530c2d4 Command failed on smithi122 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to- stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug- filestore 20 --debug-bluestore 20" ceph_test_objectstore --gtest_filter=-*/3 --gtest_catch_exceptions=0\'' [5365438] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:21:38 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365438/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365438/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=bfa967c325ff447a9ab3580f6c14ee43 Command failed (workunit test rados/test.sh) on smithi025 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365448] rados/singleton-bluestore/{all/cephtool msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:32:15 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365448/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365448/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=faa82759ebf04786a7e1c495225c6298 Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi094 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5365468] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:25:44 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365468/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365468/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=0a23f969ff23448689fa75eb597f0b6b Command failed (workunit test rados/test.sh) on smithi074 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365469] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:28:06 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365469/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365469/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=729d326fea08448e83132a368e844e1b Command failed (workunit test rados/test.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365476] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 01:00:53 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365476/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365476/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=d6ffae6ebb614d018e73bb15716bceff wait_for_clean: failed before timeout expired [5365482] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:24:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365482/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365482/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=c4b110b368484c15bf061a4f3ea7d6c5 reached maximum tries (180) after waiting for 180 seconds [5365483] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 02:08:11 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365483/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365483/ "2020-08-21T16:03:52.215934+0000 osd.4 (osd.4) 36 : cluster [ERR] 3.3 shard 6 3:c134934b:::benchmark_data_smithi070_33744_object50026:head : missing" in cluster log [5365509] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:30:44 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365509/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365509/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=01bed76afa09476f992de985d5319f66 Command failed (workunit test rados/test.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365510] rados/singleton/{all/thrash_cache_writeback_proxy_none msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:15:13 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365510/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365510/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5365525] rados/singleton-bluestore/{all/cephtool msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:33:09 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365525/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365525/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=a86aea934cd840a29a2f1ec8977e66b9 Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi060 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [5365528] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:32:02 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365528/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365528/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=b8e60d57c17f49be93e9576c71a75544 Command failed (workunit test rados/test.sh) on smithi186 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365531] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:21:39 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365531/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365531/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=0137eade0c644363b87daff765d40cc4 Command failed (workunit test rados/test.sh) on smithi089 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5365537] rados/singleton-nomsgr/{all/ceph-kvstore-tool rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:08:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365537/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365537/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=1e3e32ec43284b61a6ac8c1ddab55cf3 Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi074 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' Dead ================================================================= [5365215] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/set-chunk-promote-flush} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365215/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365215/ [5365321] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/dashboard} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365321/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365321/ [5365325] rados/cephadm/upgrade/{1-start 2-start-upgrade 3-wait distro$/{centos_8.0} fixed-2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365325/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365325/ [5365360] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365360/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365360/ [5365495] rados/cephadm/upgrade/{1-start 2-start-upgrade 3-wait distro$/{ubuntu_latest} fixed-2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365495/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365495/ Pass ================================================================= [5365201] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read} time: 00:24:50 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365201/ [5365202] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 start} time: 00:14:57 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365202/ [5365203] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/failover} time: 00:15:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365203/ [5365204] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:17:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365204/ [5365205] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:25:52 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365205/ [5365206] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:34 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365206/ [5365207] rados/singleton/{all/admin-socket msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:07:48 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365207/ [5365208] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:25 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365208/ [5365209] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_osdmap_prune} time: 00:52:20 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365209/ [5365210] rados/perf/{ceph objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:11:12 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365210/ [5365211] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:11:21 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365211/ [5365212] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/dashboard} time: 01:04:32 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365212/ [5365213] rados/objectstore/{backends/filejournal supported-random-distro$/{ubuntu_latest}} time: 00:08:23 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365213/ [5365214] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_orch_cli} time: 00:18:49 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365214/ [5365216] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:32 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365216/ [5365217] rados/singleton/{all/deduptool msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:08:51 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365217/ [5365218] rados/singleton-nomsgr/{all/ceph-post-file rados supported-random-distro$/{ubuntu_latest}} time: 00:07:08 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365218/ [5365219] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:16:57 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365219/ [5365221] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:22:21 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365221/ [5365222] rados/cephadm/smoke/{distro/rhel_latest fixed-2 start} time: 00:13:33 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365222/ [5365224] rados/singleton/{all/divergent_priors msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:13:13 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365224/ [5365225] rados/multimon/{clusters/21 msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:24:13 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365225/ [5365226] rados/perf/{ceph objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:11:29 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365226/ [5365227] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:23:30 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365227/ [5365228] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:12:55 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365228/ [5365229] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:07 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365229/ [5365230] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} time: 03:08:36 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365230/ [5365232] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:22:39 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365232/ [5365233] rados/standalone/{supported-random-distro$/{centos_8} workloads/mgr} time: 00:11:16 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365233/ [5365234] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:20:40 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365234/ [5365235] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_adoption} time: 00:09:50 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365235/ [5365236] rados/singleton/{all/divergent_priors2 msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:09:07 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365236/ [5365237] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:12:58 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365237/ [5365238] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:49 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365238/ [5365239] rados/singleton-nomsgr/{all/export-after-evict rados supported-random-distro$/{ubuntu_latest}} time: 00:07:37 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365239/ [5365240] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:22:41 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365240/ [5365241] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged msgr/async start tasks/rados_python} time: 00:24:10 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365241/ [5365242] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/rados_mon_workunits} time: 00:30:03 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365242/ [5365243] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:23:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365243/ [5365244] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:13:15 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365244/ [5365245] rados/singleton/{all/dump-stuck msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:08:58 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365245/ [5365246] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 start} time: 00:13:01 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365246/ [5365247] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_striper} time: 00:12:45 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365247/ [5365248] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:48 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365248/ [5365249] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:10 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365249/ [5365250] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8}} time: 02:22:59 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365250/ [5365251] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:19:21 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365251/ [5365252] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:29:04 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365252/ [5365253] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:12:05 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365253/ [5365254] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:24:35 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365254/ [5365255] rados/singleton/{all/ec-lost-unfound msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:49:36 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365255/ [5365256] rados/cephadm/workunits/{distro/centos_latest task/test_cephadm} time: 00:14:05 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365256/ [5365257] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:34:11 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365257/ [5365258] rados/singleton-nomsgr/{all/full-tiering rados supported-random-distro$/{ubuntu_latest}} time: 00:07:47 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365258/ [5365261] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:56 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365261/ [5365262] rados/perf/{ceph objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:12:25 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365262/ [5365263] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:07 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365263/ [5365264] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:12:23 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365264/ [5365265] rados/singleton/{all/erasure-code-nonregression msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:07:45 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365265/ [5365266] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 start} time: 00:14:11 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365266/ [5365267] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:34:55 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365267/ [5365268] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects} time: 00:20:02 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365268/ [5365269] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:33 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365269/ [5365270] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:32:12 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365270/ [5365271] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/module_selftest} time: 00:31:12 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365271/ [5365272] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:11:58 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365272/ [5365273] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_big} time: 00:31:07 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365273/ [5365275] rados/singleton/{all/lost-unfound-delete msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:30:13 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365275/ [5365276] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/snaps-few-objects} time: 00:28:23 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365276/ [5365277] rados/singleton-nomsgr/{all/health-warnings rados supported-random-distro$/{rhel_8}} time: 00:13:37 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365277/ [5365278] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_cephadm_repos} time: 00:04:44 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365278/ [5365279] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:12:05 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365279/ [5365280] rados/perf/{ceph objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:11:59 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365280/ [5365281] rados/standalone/{supported-random-distro$/{rhel_8} workloads/misc} time: 00:28:22 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365281/ [5365282] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:07 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365282/ [5365283] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged msgr/async-v2only start tasks/rados_python} time: 00:23:52 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365283/ [5365284] rados/singleton/{all/lost-unfound msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:47:53 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365284/ [5365285] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:26:52 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365285/ [5365286] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/dashboard} time: 00:58:28 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365286/ [5365287] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{ubuntu_latest}} time: 02:08:34 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365287/ [5365288] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 start} time: 00:12:57 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365288/ [5365289] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-pool-snaps} time: 00:25:01 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365289/ [5365290] rados/multimon/{clusters/6 msgr-failures/few msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:07:33 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365290/ [5365291] rados/singleton/{all/max-pg-per-osd.from-mon msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:07:26 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365291/ [5365292] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:47 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365292/ [5365293] rados/singleton-nomsgr/{all/large-omap-object-warnings rados supported-random-distro$/{centos_8}} time: 00:08:51 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365293/ [5365294] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-snaps-balanced} time: 00:26:23 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365294/ [5365295] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:12 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365295/ [5365297] rados/perf/{ceph objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:10:55 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365297/ [5365300] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:56 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365300/ [5365301] rados/cephadm/workunits/{distro/centos_latest task/test_orch_cli} time: 00:19:02 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365301/ [5365302] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/progress} time: 00:15:40 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365302/ [5365303] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mix} time: 00:24:45 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365303/ [5365304] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:31:02 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365304/ [5365305] rados/singleton/{all/max-pg-per-osd.from-primary msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:12:10 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365305/ [5365306] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:20:59 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365306/ [5365309] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache} time: 00:11:45 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365309/ [5365310] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/pool-create-delete} time: 00:18:37 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365310/ [5365311] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{ubuntu_latest}} time: 00:12:26 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365311/ [5365312] rados/singleton/{all/max-pg-per-osd.from-replica msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:12:40 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365312/ [5365313] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/dedup_tier} time: 00:10:15 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365313/ [5365314] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 start} time: 00:12:09 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365314/ [5365315] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:24 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365315/ [5365316] rados/singleton-nomsgr/{all/lazy_omap_stats_output rados supported-random-distro$/{rhel_8}} time: 00:12:26 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365316/ [5365317] rados/perf/{ceph objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:10:27 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365317/ [5365318] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:33:07 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365318/ [5365319] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:11:09 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365319/ [5365320] rados/singleton/{all/mon-auth-caps msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:09:10 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365320/ [5365322] rados/objectstore/{backends/fusestore supported-random-distro$/{rhel_8}} time: 00:10:52 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365322/ [5365324] rados/multimon/{clusters/9 msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:13:25 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365324/ [5365326] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:35:43 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365326/ [5365327] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:16 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365327/ [5365329] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:29 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365329/ [5365330] rados/standalone/{supported-random-distro$/{centos_8} workloads/mon} time: 00:45:10 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365330/ [5365331] rados/singleton/{all/mon-config-key-caps msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:10:27 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365331/ [5365332] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:57:23 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365332/ [5365333] rados/cephadm/workunits/{distro/centos_latest task/test_adoption} time: 00:09:03 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365333/ [5365334] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/prometheus} time: 00:14:18 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365334/ [5365336] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:26:49 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365336/ [5365337] rados/singleton-nomsgr/{all/librados_hello_world rados supported-random-distro$/{rhel_8}} time: 00:13:47 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365337/ [5365338] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged msgr/async start tasks/rados_python} time: 00:21:13 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365338/ [5365339] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:49:11 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365339/ [5365340] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:23:23 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365340/ [5365341] rados/singleton/{all/mon-config-keys msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:13:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365341/ [5365342] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_cephadm} time: 00:18:51 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365342/ [5365343] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:14:05 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365343/ [5365344] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_5925} time: 00:08:44 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365344/ [5365345] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/fastclose rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:29:40 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365345/ [5365346] rados/cephadm/smoke/{distro/centos_latest fixed-2 start} time: 00:11:52 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365346/ [5365347] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:11:29 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365347/ [5365348] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:31 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365348/ [5365349] rados/singleton/{all/mon-config msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:08:56 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365349/ [5365350] rados/perf/{ceph objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/sample_fio} time: 00:15:22 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365350/ [5365351] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:10:40 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365351/ [5365352] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:12:50 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365352/ [5365353] rados/singleton-nomsgr/{all/msgr rados supported-random-distro$/{ubuntu_latest}} time: 00:16:09 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365353/ [5365354] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/readwrite} time: 00:15:02 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365354/ [5365356] rados/multimon/{clusters/21 msgr-failures/few msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:07:49 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365356/ [5365357] rados/singleton/{all/osd-backfill msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:15:38 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365357/ [5365358] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/set-chunk-promote-flush} time: 00:15:20 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365358/ [5365359] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:55 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365359/ [5365361] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{rhel_8}} time: 00:16:34 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365361/ [5365362] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:19:18 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365362/ [5365363] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:51 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365363/ [5365364] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/valgrind} time: 04:11:46 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365364/ [5365365] rados/cephadm/workunits/{distro/centos_latest task/test_cephadm_repos} time: 00:05:16 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365365/ [5365366] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/workunits} time: 00:09:14 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365366/ [5365367] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/set-chunks-read} time: 00:15:15 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365367/ [5365368] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:30:25 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365368/ [5365369] rados/singleton/{all/osd-recovery-incomplete msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:21:57 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365369/ [5365370] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 start} time: 00:15:05 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365370/ [5365371] rados/perf/{ceph objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:10:51 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365371/ [5365372] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} time: 00:20:26 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365372/ [5365374] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:38:35 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365374/ [5365375] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:20:05 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365375/ [5365378] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} time: 00:27:21 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365378/ [5365379] rados/singleton/{all/osd-recovery msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:13:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365379/ [5365380] rados/standalone/{supported-random-distro$/{rhel_8} workloads/osd} time: 03:07:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365380/ [5365381] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged msgr/async-v2only start tasks/rados_python} time: 00:24:38 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365381/ [5365382] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:22 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365382/ [5365384] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/repair_test} time: 00:20:13 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365384/ [5365385] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_orch_cli} time: 00:18:45 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365385/ [5365386] rados/singleton/{all/peer msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:07:57 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365386/ [5365387] rados/perf/{ceph objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:22:22 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365387/ [5365388] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:31:39 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365388/ [5365389] rados/cephadm/smoke/{distro/rhel_latest fixed-2 start} time: 00:22:03 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365389/ [5365390] rados/multimon/{clusters/3 msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:10:49 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365390/ [5365391] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:23:14 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365391/ [5365392] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:17 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365392/ [5365393] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized} time: 00:34:47 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365393/ [5365394] rados/singleton-nomsgr/{all/osd_stale_reads rados supported-random-distro$/{ubuntu_latest}} time: 00:17:48 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365394/ [5365395] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/crash} time: 00:13:36 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365395/ [5365396] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:12:37 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365396/ [5365397] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:04 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365397/ [5365398] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/lockdep} time: 00:10:45 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365398/ [5365399] rados/singleton/{all/pg-autoscaler msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:11:01 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365399/ [5365400] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:03:14 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365400/ [5365401] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{rhel_8}} time: 00:26:47 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365401/ [5365402] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:27:24 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365402/ [5365403] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-radosbench} time: 01:50:30 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365403/ [5365404] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_adoption} time: 00:10:08 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365404/ [5365405] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:04 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365405/ [5365406] rados/perf/{ceph objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:27:47 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365406/ [5365407] rados/singleton/{all/pg-removal-interruption msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:08:20 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365407/ [5365410] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_mon_osdmap_prune} time: 00:43:59 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365410/ [5365411] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/scrub_test} time: 00:14:46 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365411/ [5365413] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 start} time: 00:12:56 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365413/ [5365414] rados/singleton-nomsgr/{all/pool-access rados supported-random-distro$/{rhel_8}} time: 00:12:14 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365414/ [5365415] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:47 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365415/ [5365416] rados/singleton/{all/radostool msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:16:45 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365416/ [5365417] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-agent-big} time: 00:14:15 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365417/ [5365418] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:52 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365418/ [5365419] rados/cephadm/workunits/{distro/centos_latest task/test_cephadm} time: 00:14:36 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365419/ [5365420] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-small} time: 00:13:08 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365420/ [5365421] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:21 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365421/ [5365422] rados/perf/{ceph objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:12:47 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365422/ [5365423] rados/multimon/{clusters/6 msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:11:38 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365423/ [5365424] rados/singleton/{all/random-eio msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:34:10 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365424/ [5365425] rados/standalone/{supported-random-distro$/{centos_8} workloads/scrub} time: 01:31:25 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365425/ [5365426] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/failover} time: 00:12:18 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365426/ [5365427] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged msgr/async-v1only start tasks/rados_python} time: 00:23:38 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365427/ [5365428] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:31:12 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365428/ [5365429] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:57 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365429/ [5365431] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 start} time: 00:13:53 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365431/ [5365432] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced} time: 00:24:30 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365432/ [5365433] rados/singleton-nomsgr/{all/recovery-unfound-found rados supported-random-distro$/{rhel_8}} time: 00:31:39 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365433/ [5365434] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-pool-snaps} time: 00:27:18 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365434/ [5365435] rados/singleton/{all/rebuild-mondb msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:18:21 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365435/ [5365436] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:05:28 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365436/ [5365439] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:25:44 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365439/ [5365440] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:11:58 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365440/ [5365441] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} time: 00:28:02 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365441/ [5365442] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:13:09 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365442/ [5365443] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/sync workloads/pool-create-delete} time: 00:16:49 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365443/ [5365444] rados/singleton/{all/recovery-preemption msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:22:40 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365444/ [5365445] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:32:10 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365445/ [5365446] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_cephadm_repos} time: 00:04:38 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365446/ [5365447] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:28:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365447/ [5365449] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:41:34 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365449/ [5365450] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root msgr/async-v1only start tasks/rados_python} time: 00:21:01 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365450/ [5365451] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache} time: 00:13:50 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365451/ [5365452] rados/singleton/{all/resolve_stuck_peering msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:09:06 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365452/ [5365453] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:53 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365453/ [5365454] rados/singleton-nomsgr/{all/version-number-sanity rados supported-random-distro$/{centos_8}} time: 00:09:02 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365454/ [5365455] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 start} time: 00:12:51 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365455/ [5365456] rados/multimon/{clusters/9 msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:07:38 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365456/ [5365457] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/dedup_tier} time: 00:14:37 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365457/ [5365458] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/insights} time: 00:12:24 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365458/ [5365459] rados/perf/{ceph objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365459/ [5365460] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365460/ [5365461] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:39 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365461/ [5365462] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/lockdep} time: 00:16:26 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365462/ [5365463] rados/singleton/{all/test-crash msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:09:59 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365463/ [5365464] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} time: 00:36:41 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365464/ [5365465] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:18:48 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365465/ [5365466] rados/cephadm/workunits/{distro/centos_latest task/test_orch_cli} time: 00:18:15 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365466/ [5365467] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read} time: 00:20:12 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365467/ [5365470] rados/singleton/{all/test_envlibrados_for_rocksdb msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:15:13 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365470/ [5365471] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:03:28 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365471/ [5365472] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} time: 00:08:44 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365472/ [5365473] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:09:22 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365473/ [5365474] rados/singleton-nomsgr/{all/admin_socket_output rados supported-random-distro$/{centos_8}} time: 00:15:43 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365474/ [5365475] rados/standalone/{supported-random-distro$/{rhel_8} workloads/crush} time: 00:14:30 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365475/ [5365477] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:18:50 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365477/ [5365478] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} time: 01:01:36 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365478/ [5365479] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_5925} time: 00:10:04 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365479/ [5365480] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{ubuntu_latest}} time: 00:11:51 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365480/ [5365481] rados/perf/{ceph objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:11:01 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365481/ [5365484] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:22 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365484/ [5365485] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 start} time: 00:11:52 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365485/ [5365486] rados/singleton/{all/thrash-backfill-full msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 01:31:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365486/ [5365487] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:32:52 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365487/ [5365488] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:05 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365488/ [5365489] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect} time: 00:11:45 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365489/ [5365490] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:11:19 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365490/ [5365491] rados/multimon/{clusters/21 msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:11:14 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365491/ [5365492] rados/singleton/{all/thrash-eio msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:30:57 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365492/ [5365493] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/module_selftest} time: 00:27:10 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365493/ [5365494] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} time: 00:15:26 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365494/ [5365496] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_python} time: 00:18:28 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365496/ [5365497] rados/singleton-nomsgr/{all/balancer rados supported-random-distro$/{rhel_8}} time: 00:13:12 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365497/ [5365498] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:34:35 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365498/ [5365499] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/valgrind} time: 00:27:37 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365499/ [5365500] rados/perf/{ceph objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:11:19 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365500/ [5365501] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:17:27 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365501/ [5365502] rados/cephadm/workunits/{distro/centos_latest task/test_adoption} time: 00:08:38 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365502/ [5365503] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:21:49 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365503/ [5365504] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:23:28 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365504/ [5365505] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root msgr/async start tasks/rados_python} time: 00:26:33 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365505/ [5365506] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/set-chunk-promote-flush} time: 00:15:18 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365506/ [5365507] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_cephadm} time: 00:18:54 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365507/ [5365508] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:16:24 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365508/ [5365511] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/dashboard} time: 01:07:16 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365511/ [5365512] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8}} time: 00:12:59 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365512/ [5365513] rados/singleton-nomsgr/{all/cache-fs-trunc rados supported-random-distro$/{centos_8}} time: 00:08:39 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365513/ [5365514] rados/perf/{ceph objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:13:07 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365514/ [5365515] rados/cephadm/smoke/{distro/centos_latest fixed-2 start} time: 00:25:35 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365515/ [5365516] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:32:48 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365516/ [5365517] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:24:04 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365517/ [5365518] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:22:48 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365518/ [5365519] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:29 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365519/ [5365520] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:10:43 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365520/ [5365521] rados/singleton/{all/watch-notify-same-primary msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:08:11 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365521/ [5365522] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:19:15 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365522/ [5365523] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:19:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365523/ [5365524] rados/standalone/{supported-random-distro$/{centos_8} workloads/erasure-code} time: 00:36:20 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365524/ [5365526] rados/multimon/{clusters/3 msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:09:42 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365526/ [5365527] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:14:42 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365527/ [5365529] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:24:00 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365529/ [5365530] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:47 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365530/ [5365532] rados/singleton/{all/admin-socket msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:07:11 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365532/ [5365533] rados/cephadm/workunits/{distro/centos_latest task/test_cephadm_repos} time: 00:04:59 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365533/ [5365534] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:11:46 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365534/ [5365535] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:30 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365535/ [5365536] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:33:41 info: http://pulpito.ceph.com/kchai-2020-08-21_13:05:18-rados-wip-kefu-testing-2020-08-21-1823-distro-basic-smithi/5365536/