Found 309 jobs Ignoring malparsed backtrace: ceph version 16.0.0-5533-g6f1dc29 (6f1dc2994c49254974a25ca5a2b36608449c133d) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7efc31812538] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55728966dc96] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-15T15:36:59.097 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-15T15:36:59.406 INFO:teuthology.orchestra.run.smithi067.stdout: Ignoring malparsed backtrace: 2020-09-15T15:38:21.439 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-09-15T15:38:21.440 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-15T15:38:21.456 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-09-15T15:40:29.258 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-09-15T15:40:29.259 DEBUG:teuthology.run_tasks:Unwinding manager selinux , 2020-09-15T15:40:29.345 INFO:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log Ignoring malparsed backtrace: node=self.hostname, label=self.label , teuthology.exceptions.CommandFailedError: Command failed on smithi093 with status 1: 'sudo cp /usr/bin/ceph_test_rados.debug /tmp/tmp.FI2ZhzHj1i' , 2020-09-15T15:40:51.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload Ignoring malparsed backtrace: ceph version 16.0.0-5533-g6f1dc29 (6f1dc2994c49254974a25ca5a2b36608449c133d) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7efc31812538] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55728966dc96] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-15T15:36:59.097 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-15T15:36:59.406 INFO:teuthology.orchestra.run.smithi067.stdout: Ignoring malparsed backtrace: 2020-09-15T15:38:21.439 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-09-15T15:38:21.440 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-15T15:38:21.456 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-09-15T15:40:29.258 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-09-15T15:40:29.259 DEBUG:teuthology.run_tasks:Unwinding manager selinux , 2020-09-15T15:40:29.345 INFO:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log Ignoring malparsed backtrace: node=self.hostname, label=self.label , teuthology.exceptions.CommandFailedError: Command failed on smithi093 with status 1: 'sudo cp /usr/bin/ceph_test_rados.debug /tmp/tmp.FI2ZhzHj1i' , 2020-09-15T15:40:51.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload Ignoring malparsed backtrace: ceph version 16.0.0-5533-g6f1dc29 (6f1dc2994c49254974a25ca5a2b36608449c133d) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7efc31812538] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55728966dc96] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-15T15:36:59.097 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-15T15:36:59.406 INFO:teuthology.orchestra.run.smithi067.stdout: Ignoring malparsed backtrace: 2020-09-15T15:38:21.439 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-09-15T15:38:21.440 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-15T15:38:21.456 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-09-15T15:40:29.258 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-09-15T15:40:29.259 DEBUG:teuthology.run_tasks:Unwinding manager selinux , 2020-09-15T15:40:29.345 INFO:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log Ignoring malparsed backtrace: node=self.hostname, label=self.label , teuthology.exceptions.CommandFailedError: Command failed on smithi093 with status 1: 'sudo cp /usr/bin/ceph_test_rados.debug /tmp/tmp.FI2ZhzHj1i' , 2020-09-15T15:40:51.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload Found 6 distinct failure reasons Timeout 3h running clone.client.0/qa/workunits/cephtool/test.sh 4 jobs: ['5436731', '5436801', '5436665', '5436596'] suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool'] suites union: ['msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 1 jobs: ['5436734'] suites: ['msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{rhel_8}}'] Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 4 jobs: ['5436706', '5436632', '5436786', '5436549'] suites intersection: ['task/test_orch_cli}'] suites union: ['rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_orch_cli}'] Failure: Test failure: test_osd_cannot_recover (tasks.mgr.test_progress.TestProgress) 2 jobs: ['5436782', '5436575'] suites intersection: ['debug/mgr', 'rados/mgr/{clusters/{2-node-mgr}', 'tasks/progress}'] suites union: ['debug/mgr', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/progress}'] Timeout 120 running ceph --cluster ceph --log-early quorum_status 1 jobs: ['5436737'] suites: ['msgr-failures/many', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-stupid', 'rados', 'rados/multimon/{clusters/3', 'supported-random-distro$/{rhel_8}', 'tasks/mon_recovery}'] Failure: reached maximum tries (800) after waiting for 4800 seconds 1 jobs: ['5436826'] suites: ['clusters/{fixed-2', 'fast/normal', 'msgr-failures/fastclose', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-radosbench}']