Found 337 jobs Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , Ignoring malparsed backtrace: ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1ef6d36157] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x557ea7a8fbc1] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-21T16:21:48.145 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-21T16:21:48.415 INFO:teuthology.orchestra.run.smithi188.stdout: Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait , proc.wait() Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status Ignoring malparsed backtrace: 2020-08-21T16:26:05.375 INFO:teuthology.orchestra.run.smithi074.stdout: ceph-mgr-cephadm* cephadm* , 2020-08-21T16:26:05.564 INFO:teuthology.orchestra.run.smithi188.stdout:0 upgraded, 0 newly installed, 2 to remove and 18 not upgraded. , 2020-08-21T16:26:05.565 INFO:teuthology.orchestra.run.smithi188.stdout:After this operation, 494 kB disk space will be freed. Ignoring malparsed backtrace: (Reading database ... 60% , (Reading database ... 65% , (Reading database ... 70% Ignoring malparsed backtrace: (Reading database ... 100% , (Reading database ... 109770 files and directories currently installed.) , 2020-08-21T16:26:10.070 INFO:teuthology.orchestra.run.smithi074.stdout:Purging configuration files for ceph-mds (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:12.507 INFO:teuthology.orchestra.run.smithi188.stdout:(Reading database ... , (Reading database ... 5% , (Reading database ... 10% Ignoring malparsed backtrace: 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py , 2020-08-21T16:26:16.747 INFO:teuthology.orchestra.run.smithi074.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-olefile python3-paste python3-pastedeploy python3-pastescript , 2020-08-21T16:26:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py Ignoring malparsed backtrace: (Reading database ... 75% , (Reading database ... 80% , (Reading database ... 85% Ignoring malparsed backtrace: 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch , 2020-08-21T16:26:31.313 INFO:teuthology.orchestra.run.smithi074.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako , 2020-08-21T16:26:31.314 INFO:teuthology.orchestra.run.smithi074.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy Ignoring malparsed backtrace: (Reading database ... 50% , (Reading database ... 55% , (Reading database ... 60% Ignoring malparsed backtrace: 2020-08-21T16:26:35.005 INFO:teuthology.orchestra.run.smithi188.stdout:The following packages were automatically installed and are no longer required: , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core jq libjq1 libleveldb1v5 liboath0 libonig4 librabbitmq4 , 2020-08-21T16:26:35.006 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr Ignoring malparsed backtrace: (Reading database ... 15% , (Reading database ... 20% , (Reading database ... 25% Ignoring malparsed backtrace: (Reading database ... 10% , (Reading database ... 15% , (Reading database ... 20% Ignoring malparsed backtrace: 2020-08-21T16:26:40.004 INFO:teuthology.orchestra.run.smithi074.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:40.005 INFO:teuthology.orchestra.run.smithi074.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: 2020-08-21T16:26:41.162 INFO:teuthology.orchestra.run.smithi188.stdout:Removing librgw2 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.462 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... , 2020-08-21T16:26:41.625 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libradosstriper1 (16.0.0-4581-gfcd27822606-1bionic) ... Ignoring malparsed backtrace: 2020-08-21T16:26:47.834 INFO:teuthology.orchestra.run.smithi074.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-tz python3-uritemplate python3-waitress python3-webencodings , 2020-08-21T16:26:47.835 INFO:teuthology.orchestra.run.smithi074.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug seabios Ignoring malparsed backtrace: 2020-08-21T16:26:48.515 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes , 2020-08-21T16:26:48.516 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe Ignoring malparsed backtrace: (Reading database ... 90% , (Reading database ... 95% , (Reading database ... 100% Ignoring malparsed backtrace: 2020-08-21T16:26:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... , 2020-08-21T16:26:59.704 INFO:teuthology.orchestra.run.smithi074.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... , 2020-08-21T16:26:59.852 INFO:teuthology.orchestra.run.smithi188.stdout:Removing libmp3lame0:amd64 (3.100-2) ... Ignoring malparsed backtrace: 2020-08-21T16:27:21.146 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-sklearn (0.19.1-3) ... , 2020-08-21T16:27:21.247 INFO:teuthology.orchestra.run.smithi074.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... , 2020-08-21T16:27:21.685 INFO:teuthology.orchestra.run.smithi188.stdout:Removing python3-scipy (0.19.1-2ubuntu1) ... Ignoring malparsed backtrace: 2020-08-21T16:27:44.467 INFO:teuthology.task.install:Removing ceph sources lists , 2020-08-21T16:27:44.469 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is , 2020-08-21T16:27:44.470 INFO:teuthology.orchestra.run.smithi074:> sudo lsb_release -is Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , Found 10 distinct failure reasons Failure: reached maximum tries (180) after waiting for 180 seconds 4 jobs: ['5365482', '5365231', '5365409', '5365307'] suites intersection: ['ceph', 'clusters/{openstack', 'distro$/{centos_7.6}', 'rados', 'thrashosds-health', 'three-plus-one}'] suites union: ['1-install/luminous', '1-install/luminous-v1only', '1-install/mimic', '1-install/nautilus', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'distro$/{centos_7.6}', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}', 'workloads/test_rbd_api}'] Dead: b'2020-08-22T02:02:18.058 DEBUG:teuthology.exit:Finished running handlers' 5 jobs: ['5365321', '5365215', '5365325', '5365360', '5365495'] suites intersection: [] suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '2-start-upgrade', '3-wait', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'debug/mgr', 'distro$/{centos_8.0}', 'distro$/{ubuntu_latest}', 'fixed-2}', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/cephadm/upgrade/{1-start', 'rados/dashboard/{clusters/{2-node-mgr}', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/dashboard}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/set-chunk-promote-flush}'] Failure: "2020-08-21T14:43:38.297138+0000 osd.2 (osd.2) 40 : cluster [ERR] 2.8 shard 2 2:10fa4044:::benchmark_data_smithi149_15659_object110629:head : missing" in cluster log 5 jobs: ['5365376', '5365412', '5365274', '5365483', '5365335'] suites intersection: ['rados'] suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'fast/normal', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/multi-backfill-reject', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-agent-big}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/radosbench}'] Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi159 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' ceph version 16.0.0-4581-gfcd27822606 (fcd27822606f23182148cf933d868bb24cade17f) pacific (dev) 1: (()+0x12dc0) [0x7f1b5eef7dc0] 2: (BlueFS::_allocate(unsigned char, unsigned long, bluefs_fnode_t*)+0x873) [0x55c90625b463] 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x13e) [0x55c90625b73e] 4: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x110) [0x55c90625d430] 5: (BlueFS::_fsync(BlueFS::FileWriter*, std::unique_lock&)+0x159) [0x55c90625fe79] 6: (BlueRocksWritableFile::Sync()+0x6c) [0x55c906289d2c] 7: (rocksdb::WriteStringToFile(rocksdb::Env*, rocksdb::Slice const&, std::__cxx11::basic_string, std::allocator > const&, bool)+0x107) [0x55c9064681f7] 8: (rocksdb::SetIdentityFile(rocksdb::Env*, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&)+0x114) [0x55c906473244] 9: (rocksdb::DBImpl::NewDB()+0x172) [0x55c9063a3162] 10: (()+0x5ad286) [0x55c90641a286] 11: (rocksdb::RepairDB(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Options const&)+0x13d) [0x55c90641ca7d] 12: (RocksDBStore::repair(std::ostream&)+0x43f) [0x55c90631e85f] 13: (main()+0x2a6) [0x55c90606ae86] 14: (__libc_start_main()+0xf3) [0x7f1b5d59a873] 15: (_start()+0x2e) [0x55c906094a1e] 6 jobs: ['5365373', '5365448', '5365298', '5365525', '5365537', '5365223'] suites intersection: ['rados'] suites union: ['msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test rados/test.sh) on smithi065 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcd27822606f23182148cf933d868bb24cade17f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 16 jobs: ['5365323', '5365531', '5365509', '5365438', '5365260', '5365296', '5365430', '5365408', '5365308', '5365469', '5365528', '5365355', '5365220', '5365468', '5365377', '5365328'] suites intersection: [] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/9-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'd-thrash/none', 'fixed-2', 'mode/packaged', 'mode/root', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/cephadm/with-work/{distro/centos_8.0', 'rados/cephadm/with-work/{distro/centos_latest', 'rados/cephadm/with-work/{distro/rhel_8.0', 'rados/cephadm/with-work/{distro/rhel_latest', 'rados/cephadm/with-work/{distro/ubuntu_18.04', 'rados/monthrash/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'start', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests', 'tasks/rados_api_tests}', 'thrashers/careful', 'thrashers/default', 'thrashers/many', 'thrashers/one', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}', 'workloads/rados_api_tests}'] Failure: Command failed on smithi107 with status 1: 'sudo yum -y install ceph-test' 2 jobs: ['5365383', '5365259'] suites intersection: ['rados', 'supported-random-distro$/{centos_8}'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/multimon/{clusters/3', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'tasks/mon_clock_no_skews}', 'thrashers/none', 'thrashosds-health', 'workloads/small-objects}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 1 jobs: ['5365510'] suites: ['msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}'] Failure: wait_for_clean: failed before timeout expired 1 jobs: ['5365476'] suites: ['1-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'bluestore-bitmap', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}'] Failure: Command failed on smithi122 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-filestore 20 --debug-bluestore 20" ceph_test_objectstore --gtest_filter=-*/3 --gtest_catch_exceptions=0\'' 1 jobs: ['5365437'] suites: ['rados/objectstore/{backends/objectstore', 'supported-random-distro$/{centos_8}}'] Timeout 120 running ceph --cluster ceph --log-early osd last-stat-seq osd.2 1 jobs: ['5365299'] suites: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-pool-snaps-few-objects-overwrites}']