Found 367 jobs Ignoring malparsed backtrace: ceph version 16.0.0-8048-ge8dd40f3 (e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf98eca662] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55e596bcb306] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-09T17:03:40.944 INFO:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-09T17:03:41.251 INFO:teuthology.orchestra.run.smithi055.stdout: Ignoring malparsed backtrace: 2020-12-09T17:04:48.989 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-12-09T17:04:48.990 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-09T17:04:49.010 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-12-09T17:11:45.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-09T17:11:45.997 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-09T17:11:45.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8048-ge8dd40f3 (e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf98eca662] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55e596bcb306] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-09T17:03:40.944 INFO:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-09T17:03:41.251 INFO:teuthology.orchestra.run.smithi055.stdout: Ignoring malparsed backtrace: 2020-12-09T17:04:48.989 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-12-09T17:04:48.990 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-09T17:04:49.010 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-12-09T17:11:45.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-09T17:11:45.997 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-09T17:11:45.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8048-ge8dd40f3 (e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf98eca662] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55e596bcb306] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-09T17:03:40.944 INFO:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-09T17:03:41.251 INFO:teuthology.orchestra.run.smithi055.stdout: Ignoring malparsed backtrace: 2020-12-09T17:04:48.989 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-12-09T17:04:48.990 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-09T17:04:49.010 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-12-09T17:11:45.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-09T17:11:45.997 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-09T17:11:45.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8048-ge8dd40f3 (e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf98eca662] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55e596bcb306] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-09T17:03:40.944 INFO:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-09T17:03:41.251 INFO:teuthology.orchestra.run.smithi055.stdout: Ignoring malparsed backtrace: 2020-12-09T17:04:48.989 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-12-09T17:04:48.990 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-09T17:04:49.010 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-12-09T17:11:45.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-09T17:11:45.997 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-09T17:11:45.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8048-ge8dd40f3 (e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf98eca662] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55e596bcb306] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-09T17:03:40.944 INFO:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-09T17:03:41.251 INFO:teuthology.orchestra.run.smithi055.stdout: Ignoring malparsed backtrace: 2020-12-09T17:04:48.989 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-12-09T17:04:48.990 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-09T17:04:49.010 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-12-09T17:11:45.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-09T17:11:45.997 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-09T17:11:45.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8048-ge8dd40f3 (e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf98eca662] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55e596bcb306] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-09T17:03:40.944 INFO:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-09T17:03:41.251 INFO:teuthology.orchestra.run.smithi055.stdout: Ignoring malparsed backtrace: 2020-12-09T17:04:48.989 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-12-09T17:04:48.990 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-09T17:04:49.010 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-12-09T17:11:45.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-09T17:11:45.997 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-09T17:11:45.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8048-ge8dd40f3 (e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf98eca662] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55e596bcb306] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-09T17:03:40.944 INFO:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-09T17:03:41.251 INFO:teuthology.orchestra.run.smithi055.stdout: Ignoring malparsed backtrace: 2020-12-09T17:04:48.989 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-12-09T17:04:48.990 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-09T17:04:49.010 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-12-09T17:11:45.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-09T17:11:45.997 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-09T17:11:45.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8048-ge8dd40f3 (e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faf98eca662] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55e596bcb306] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-09T17:03:40.944 INFO:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-09T17:03:41.251 INFO:teuthology.orchestra.run.smithi055.stdout: Ignoring malparsed backtrace: 2020-12-09T17:04:48.989 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2020-12-09T17:04:48.990 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-09T17:04:49.010 INFO:tasks.ceph.mgr.x:Stopped Ignoring malparsed backtrace: 2020-12-09T17:11:45.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-09T17:11:45.997 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-09T17:11:45.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux Found 10 distinct failure reasons Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 1 jobs: ['5695718'] suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_orch_cli}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi140 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 1 jobs: ['5695652'] suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] Failure: Command failed (workunit test cephtool/test.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2 jobs: ['5695418', '5695696'] suites intersection: ['objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}'] Failure: reached maximum tries (180) after waiting for 180 seconds 5 jobs: ['5695756', '5695482', '5695679', '5695574', '5695530'] suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_8.0}', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2 jobs: ['5695429', '5695613'] suites intersection: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] suites union: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 1 jobs: ['5695650'] suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{centos_8}}'] Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8dd40f366b9eecf8b9fd44aad6fe2b0ad80a384 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 1 jobs: ['5695478'] suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] Dead: b'2020-12-10T01:26:36.780 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['5695446'] suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] Failure: "2020-12-09T14:35:20.986337+0000 mon.b (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log 1 jobs: ['5695570'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/redirect_promote_tests}'] Timeout 120 running ceph --cluster ceph tell osd.1 flush_pg_stats 1 jobs: ['5695664'] suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}']