2020-11-20T07:22:43.409 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-20T07:22:43.410 INFO:teuthology.results:subset: '9000/10000' 2020-11-20T07:22:43.411 INFO:teuthology.results:seed: '3673' 2020-11-20T07:39:49.026 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-20T07:39:49.080 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-11-20T10:56:24.204 INFO:teuthology.results:Tests finished! gathering results... 2020-11-20T10:56:30.867 INFO:scrape:Found 39 jobs 2020-11-20T10:56:31.195 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6860-ge11b788b (e11b788b41a72cdbe2c77fab79a664262b358e1f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7fc9d02e01eb] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55606a6b49b0] 2020-11-20T10:56:31.198 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-11-20T08:00:33.466 INFO:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-20T08:00:33.807 INFO:teuthology.orchestra.run.smithi170.stdout: 2020-11-20T10:56:31.198 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2020-11-20T10:56:31.199 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-mgr-progress-global-efficiency/qa/tasks/rados.py", line 277, in task , running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get 2020-11-20T10:56:31.330 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6860-ge11b788b (e11b788b41a72cdbe2c77fab79a664262b358e1f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7fc9d02e01eb] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55606a6b49b0] 2020-11-20T10:56:31.333 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-11-20T08:00:33.466 INFO:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-20T08:00:33.807 INFO:teuthology.orchestra.run.smithi170.stdout: 2020-11-20T10:56:31.334 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2020-11-20T10:56:31.335 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-mgr-progress-global-efficiency/qa/tasks/rados.py", line 277, in task , running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get 2020-11-20T10:56:31.455 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6860-ge11b788b (e11b788b41a72cdbe2c77fab79a664262b358e1f) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7fc9d02e01eb] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55606a6b49b0] 2020-11-20T10:56:31.457 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-11-20T08:00:33.466 INFO:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-20T08:00:33.807 INFO:teuthology.orchestra.run.smithi170.stdout: 2020-11-20T10:56:31.458 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2020-11-20T10:56:31.458 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-mgr-progress-global-efficiency/qa/tasks/rados.py", line 277, in task , running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get 2020-11-20T10:56:34.879 INFO:scrape:Found 6 distinct failure reasons 2020-11-20T10:56:34.880 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-11-20T10:56:34.880 INFO:scrape:1 jobs: ['5640096'] 2020-11-20T10:56:34.880 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}'] 2020-11-20T10:56:34.881 INFO:scrape: 2020-11-20T10:56:34.881 INFO:scrape:Failure: Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4711d7b2-2b05-11eb-8c4a-001a4aab830c -- ceph orch host add smithi012' 2020-11-20T10:56:34.882 INFO:scrape:2 jobs: ['5640088', '5640103'] 2020-11-20T10:56:34.882 INFO:scrape:suites intersection: ['2-repo_digest/defaut', '3-start-upgrade', '4-wait', 'distro$/{ubuntu_18.04_podman}', 'fixed-2', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-11-20T10:56:34.883 INFO:scrape:suites union: ['2-repo_digest/defaut', '3-start-upgrade', '4-wait', 'distro$/{ubuntu_18.04_podman}', 'fixed-2', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-11-20T10:56:34.883 INFO:scrape: 2020-11-20T10:56:34.883 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/mon/caps.sh 2020-11-20T10:56:34.884 INFO:scrape:1 jobs: ['5640085'] 2020-11-20T10:56:34.884 INFO:scrape:suites: ['clusters/3-mons', 'mon_election/connectivity', 'msgr-failures/mon-delay', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/sync', 'workloads/rados_mon_workunits}'] 2020-11-20T10:56:34.885 INFO:scrape: 2020-11-20T10:56:34.885 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi086 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e11b788b41a72cdbe2c77fab79a664262b358e1f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2020-11-20T10:56:34.886 INFO:scrape:2 jobs: ['5640097', '5640077'] 2020-11-20T10:56:34.886 INFO:scrape:suites intersection: ['mon_election/classic', 'task/test_adoption}'] 2020-11-20T10:56:34.886 INFO:scrape:suites union: ['mon_election/classic', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2020-11-20T10:56:34.887 INFO:scrape: 2020-11-20T10:56:34.887 INFO:scrape:Failure: Command failed on smithi052 with status 5: 'sudo systemctl stop ceph-e58f52ae-2b03-11eb-8c4a-001a4aab830c@mon.a' 2020-11-20T10:56:34.888 INFO:scrape:6 jobs: ['5640082', '5640098', '5640094', '5640093', '5640078', '5640084'] 2020-11-20T10:56:34.888 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] 2020-11-20T10:56:34.888 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_8.0}', 'distro$/{centos_latest}', 'distro$/{rhel_latest}', 'distro$/{ubuntu_18.04}', 'distro$/{ubuntu_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-11-20T10:56:34.889 INFO:scrape: 2020-11-20T10:56:34.889 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi094 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e11b788b41a72cdbe2c77fab79a664262b358e1f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2020-11-20T10:56:34.890 INFO:scrape:3 jobs: ['5640100', '5640090', '5640079'] 2020-11-20T10:56:34.890 INFO:scrape:suites intersection: ['task/test_cephadm}'] 2020-11-20T10:56:34.890 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-11-20T10:56:34.891 INFO:scrape: