2021-01-26T00:10:50.667 INFO:root:teuthology version: 1.0.0-fe79ba1b
2021-01-26T00:10:50.671 INFO:teuthology.results:subset: '1/55000'
2021-01-26T00:10:50.671 INFO:teuthology.results:seed: '7552'
2021-01-26T02:17:19.469 INFO:root:teuthology version: 1.0.0-fe79ba1b
2021-01-26T02:17:19.512 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-01-26T14:17:40.936 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2021-01-26T14:18:14.193 INFO:scrape:Found 260 jobs
2021-01-26T14:18:14.626 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f9fa72ee0ab]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55fb2b011d60]

2021-01-26T14:18:14.628 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub
, 2021-01-26T02:21:30.261 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-01-26T02:21:30.551 INFO:teuthology.orchestra.run.smithi164.stdout:

2021-01-26T14:18:14.628 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-26T02:22:14.705 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi165.front.sepia.ceph.com
, 2021-01-26T02:22:14.706 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4
, 2021-01-26T02:22:14.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi165.front.sepia.ceph.com

2021-01-26T14:18:14.629 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-26T02:24:00.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
, 2021-01-26T02:24:00.359 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2021-01-26T02:24:00.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump

2021-01-26T14:18:14.733 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f9fa72ee0ab]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55fb2b011d60]

2021-01-26T14:18:14.735 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub
, 2021-01-26T02:21:30.261 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-01-26T02:21:30.551 INFO:teuthology.orchestra.run.smithi164.stdout:

2021-01-26T14:18:14.736 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-26T02:22:14.705 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi165.front.sepia.ceph.com
, 2021-01-26T02:22:14.706 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4
, 2021-01-26T02:22:14.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi165.front.sepia.ceph.com

2021-01-26T14:18:14.736 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-26T02:24:00.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
, 2021-01-26T02:24:00.359 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2021-01-26T02:24:00.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump

2021-01-26T14:18:14.837 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f9fa72ee0ab]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55fb2b011d60]

2021-01-26T14:18:14.839 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub
, 2021-01-26T02:21:30.261 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-01-26T02:21:30.551 INFO:teuthology.orchestra.run.smithi164.stdout:

2021-01-26T14:18:14.839 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-26T02:22:14.705 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi165.front.sepia.ceph.com
, 2021-01-26T02:22:14.706 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4
, 2021-01-26T02:22:14.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi165.front.sepia.ceph.com

2021-01-26T14:18:14.840 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-26T02:24:00.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
, 2021-01-26T02:24:00.359 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2021-01-26T02:24:00.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump

2021-01-26T14:18:19.245 INFO:scrape:Found 6 distinct failure reasons
2021-01-26T14:18:19.247 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base'
2021-01-26T14:18:19.247 INFO:scrape:1 jobs: ['5828835']
2021-01-26T14:18:19.247 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}']
2021-01-26T14:18:19.248 INFO:scrape:
2021-01-26T14:18:19.248 INFO:scrape:Dead: b'2021-01-26T14:15:33.217 DEBUG:teuthology.exit:Finished running handlers'
2021-01-26T14:18:19.249 INFO:scrape:3 jobs: ['5828854', '5828719', '5828837']
2021-01-26T14:18:19.249 INFO:scrape:suites intersection: []
2021-01-26T14:18:19.250 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/objectstore/{backends/objectstore', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/radosbench-high-concurrency}']
2021-01-26T14:18:19.250 INFO:scrape:
2021-01-26T14:18:19.251 INFO:scrape:Failure: "2021-01-26T01:08:46.748207+0000 mon.b (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log
2021-01-26T14:18:19.251 INFO:scrape:1 jobs: ['5828660']
2021-01-26T14:18:19.251 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}']
2021-01-26T14:18:19.252 INFO:scrape:
2021-01-26T14:18:19.252 INFO:scrape:Failure: Command failed on smithi028 with status 1: '/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml'
2021-01-26T14:18:19.253 INFO:scrape:1 jobs: ['5828796']
2021-01-26T14:18:19.253 INFO:scrape:suites: ['mon_election/classic', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/dmclock_default_shards', 'settings/optimized', 'ubuntu_latest', 'workloads/sample_fio}']
2021-01-26T14:18:19.254 INFO:scrape:
2021-01-26T14:18:19.254 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
2021-01-26T14:18:19.254 INFO:scrape:1 jobs: ['5828781']
2021-01-26T14:18:19.255 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']
2021-01-26T14:18:19.255 INFO:scrape:
2021-01-26T14:18:19.256 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5f17c37f78a331b7a4bf793890f9d324c64183e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
2021-01-26T14:18:19.256 INFO:scrape:1 jobs: ['5828711']
2021-01-26T14:18:19.256 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}']
2021-01-26T14:18:19.257 INFO:scrape: