Found 446 jobs Ignoring malparsed backtrace: ceph version 16.0.0-7582-g2abe3351 (2abe33517660ea12e639748e378aec66dcd6ac85) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7ff3c65f2d8d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff3c65f2f68] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-11-26T08:03:37.241 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-26T08:03:37.550 INFO:teuthology.orchestra.run.smithi086.stdout: Ignoring malparsed backtrace: 2020-11-26T08:04:59.590 INFO:teuthology.orchestra.run:waiting for 300 , 2020-11-26T08:04:59.614 INFO:tasks.ceph.mgr.x:Stopped , 2020-11-26T08:04:59.615 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: ceph version 16.0.0-7582-g2abe3351 (2abe33517660ea12e639748e378aec66dcd6ac85) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7ff3c65f2d8d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff3c65f2f68] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-11-26T08:03:37.241 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-26T08:03:37.550 INFO:teuthology.orchestra.run.smithi086.stdout: Ignoring malparsed backtrace: 2020-11-26T08:04:59.590 INFO:teuthology.orchestra.run:waiting for 300 , 2020-11-26T08:04:59.614 INFO:tasks.ceph.mgr.x:Stopped , 2020-11-26T08:04:59.615 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: ceph version 16.0.0-7582-g2abe3351 (2abe33517660ea12e639748e378aec66dcd6ac85) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7ff3c65f2d8d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff3c65f2f68] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-11-26T08:03:37.241 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-26T08:03:37.550 INFO:teuthology.orchestra.run.smithi086.stdout: Ignoring malparsed backtrace: 2020-11-26T08:04:59.590 INFO:teuthology.orchestra.run:waiting for 300 , 2020-11-26T08:04:59.614 INFO:tasks.ceph.mgr.x:Stopped , 2020-11-26T08:04:59.615 INFO:teuthology.misc:Shutting down mon daemons... Found 15 distinct failure reasons Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 1 jobs: ['5657940'] suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}'] Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 1 jobs: ['5657838'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}'] Timeout 3h running clone.client.0/qa/workunits/mon/caps.sh 3 jobs: ['5657580', '5657622', '5657876'] suites intersection: ['rados', 'rados/monthrash/{ceph', 'workloads/rados_mon_workunits}'] suites union: ['clusters/3-mons', 'clusters/9-mons', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/many', 'thrashers/one', 'thrashers/sync-many', 'workloads/rados_mon_workunits}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi179 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2abe33517660ea12e639748e378aec66dcd6ac85 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 3 jobs: ['5657987', '5657828', '5657662'] suites intersection: ['task/test_cephadm}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] Failure: reached maximum tries (180) after waiting for 180 seconds 2 jobs: ['5657788', '5657566'] suites intersection: ['2-repo_digest/defaut', '3-start-upgrade', '4-wait', 'fixed-2', 'mon_election/classic}', 'rados/cephadm/upgrade/{1-start'] suites union: ['2-repo_digest/defaut', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'rados/cephadm/upgrade/{1-start'] Failure: "2020-11-26T08:19:07.514247+0000 osd.2 (osd.2) 53 : cluster [ERR] scrub 57.b 57:d1a6be1c:test-rados-api-smithi002-23458-74::foo:24 : size 0 != clone_size 10" in cluster log 1 jobs: ['5657956'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/rados_api_tests}'] Failure: Command failed on smithi125 with status 5: 'sudo systemctl stop ceph-c07fd844-2fa9-11eb-980d-001a4aab830c@mon.a' 2 jobs: ['5657616', '5657842'] suites intersection: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'fixed-2', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] suites union: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{ubuntu_18.04}', 'distro$/{ubuntu_latest}', 'fixed-2', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2abe33517660ea12e639748e378aec66dcd6ac85 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 3 jobs: ['5657953', '5657790', '5657626'] suites intersection: ['task/test_adoption}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] Failure: "2020-11-26T10:54:19.888097+0000 osd.1 (osd.1) 176 : cluster [ERR] 5.f deep-scrub : stat mismatch, got 4/4 objects, 0/0 clones, 0/0 dirty, 0/0 omap, 0/0 pinned, 4/4 hit_set_archive, 0/0 whiteouts, 6847/0 bytes, 0/0 manifest objects, 6847/6847 hit_set_archive bytes." in cluster log 1 jobs: ['5657994'] suites: ['1-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'bluestore-bitmap', 'mon_election/connectivity', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}'] Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 1 jobs: ['5657905'] suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_orch_cli}'] Failure: Command failed on smithi162 with status 13: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' 1 jobs: ['5657843'] suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton/{all/rebuild-mondb', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test rados/test.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2abe33517660ea12e639748e378aec66dcd6ac85 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 1 jobs: ['5657857'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}'] Dead: b'2020-11-26T17:11:54.384 DEBUG:teuthology.exit:Finished running handlers' 2 jobs: ['5657576', '5657661'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'thrashosds-health}', 'validater/valgrind}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}'] Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi164 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2abe33517660ea12e639748e378aec66dcd6ac85 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2 jobs: ['5657774', '5657998'] suites intersection: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] suites union: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] Failure: '/home/ubuntu/cephtest/archive/syslog/misc.log:2020-11-26T06:27:13.083325+00:00 smithi119 bash[8542]: debug 2020-11-26T06:27:13.075+0000 7f3a927bf700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploying daemon prometheus.a ... ' in syslog 1 jobs: ['5657723'] suites: ['2-repo_digest/defaut', '3-start-upgrade', '4-wait', 'distro$/{ubuntu_18.04}', 'fixed-2', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start']