2021-10-01T06:36:33.467 INFO:root:teuthology version: 1.1.0-f08b273e
2021-10-01T06:36:33.469 INFO:teuthology.results:seed: '9668'
2021-10-01T06:40:13.820 INFO:root:teuthology version: 1.1.0-f08b273e
2021-10-01T06:40:13.825 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-10-01T07:37:17.970 INFO:teuthology.results:Tests finished! gathering results...
2021-10-01T07:37:18.093 INFO:scrape:Found 4 jobs
2021-10-01T07:37:18.280 INFO:scrape:Found 2 distinct failure reasons
2021-10-01T07:37:18.281 INFO:scrape:Crash: Command failed on smithi031 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats'
2021-10-01T07:37:18.281 INFO:scrape:ceph version 17.0.0-8011-g6939ea03 (6939ea034a21745448742eb1ff6186336a4fd48f) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7f732d146b20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x5652dd324e52]
 5: ceph-osd(+0x5d2015) [0x5652dd325015]
 6: (ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)+0x1b3c) [0x5652dd81e79c]
 7: (ECBackend::run_recovery_op(PGBackend::RecoveryHandle*, int)+0x441) [0x5652dd81fd01]
 8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x26f9) [0x5652dd5acb69]
 9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x10e3) [0x5652dd5b0763]
 10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x2f5) [0x5652dd42b845]
 11: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1d) [0x5652dd6edb3d]
 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xac8) [0x5652dd449fe8]
 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x5652ddb38f94]
 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5652ddb39a84]
 15: /lib64/libpthread.so.0(+0x814a) [0x7f732d13c14a]
 16: clone()
2021-10-01T07:37:18.282 INFO:scrape:3 jobs: ['6415947', '6415944', '6415945']
2021-10-01T07:37:18.282 INFO:scrape:suites intersection: ['rados', 'rados/singleton/{all/ec-inconsistent-hinfo']
2021-10-01T07:37:18.283 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/ec-inconsistent-hinfo', 'supported-random-distro$/{centos_8.stream}}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}']
2021-10-01T07:37:18.283 INFO:scrape:
2021-10-01T07:37:18.284 INFO:scrape:Failure: Command failed on smithi017 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-8011-g6939ea03-1focal cephadm=17.0.0-8011-g6939ea03-1focal ceph-mds=17.0.0-8011-g6939ea03-1focal ceph-mgr=17.0.0-8011-g6939ea03-1focal ceph-common=17.0.0-8011-g6939ea03-1focal ceph-fuse=17.0.0-8011-g6939ea03-1focal ceph-test=17.0.0-8011-g6939ea03-1focal ceph-volume=17.0.0-8011-g6939ea03-1focal radosgw=17.0.0-8011-g6939ea03-1focal python3-rados=17.0.0-8011-g6939ea03-1focal python3-rgw=17.0.0-8011-g6939ea03-1focal python3-cephfs=17.0.0-8011-g6939ea03-1focal python3-rbd=17.0.0-8011-g6939ea03-1focal libcephfs2=17.0.0-8011-g6939ea03-1focal libcephfs-dev=17.0.0-8011-g6939ea03-1focal librados2=17.0.0-8011-g6939ea03-1focal librbd1=17.0.0-8011-g6939ea03-1focal rbd-fuse=17.0.0-8011-g6939ea03-1focal'
2021-10-01T07:37:18.284 INFO:scrape:1 jobs: ['6415946']
2021-10-01T07:37:18.285 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/none', 'msgr/async', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/ec-inconsistent-hinfo', 'supported-random-distro$/{ubuntu_latest}}']
2021-10-01T07:37:18.286 INFO:scrape:
2021-10-01T07:37:18.286 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6415944]  rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:47:26
info:   http://pulpito.ceph.com/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/6415944/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/6415944/
sentry: https://sentry.ceph.com/organizations/ceph/?query=52ddeab0d15c4b8a9ad994ca138c4ff5

    Command failed on smithi166 with status 6: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph tell osd.3 flush_pg_stats'

[6415945]  rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:53:13
info:   http://pulpito.ceph.com/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/6415945/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/6415945/
sentry: https://sentry.ceph.com/organizations/ceph/?query=509cfd41515445ad90c9dd6c9ea08922

    Command failed on smithi118 with status 6: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph tell osd.3 flush_pg_stats'

[6415946]  rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/none msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:03:39
info:   http://pulpito.ceph.com/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/6415946/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/6415946/
sentry: https://sentry.ceph.com/organizations/ceph/?query=438288265b7f46d590557153ace62422

    Command failed on smithi017 with status 100: 'sudo
    DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="
    --force-confdef" -o Dpkg::Options::="--force-confold" install
    ceph=17.0.0-8011-g6939ea03-1focal cephadm=17.0.0-8011-g6939ea03-1focal
    ceph-mds=17.0.0-8011-g6939ea03-1focal ceph-mgr=17.0.0-8011-g6939ea03-1focal
    ceph-common=17.0.0-8011-g6939ea03-1focal ceph-
    fuse=17.0.0-8011-g6939ea03-1focal ceph-test=17.0.0-8011-g6939ea03-1focal
    ceph-volume=17.0.0-8011-g6939ea03-1focal
    radosgw=17.0.0-8011-g6939ea03-1focal
    python3-rados=17.0.0-8011-g6939ea03-1focal
    python3-rgw=17.0.0-8011-g6939ea03-1focal
    python3-cephfs=17.0.0-8011-g6939ea03-1focal
    python3-rbd=17.0.0-8011-g6939ea03-1focal
    libcephfs2=17.0.0-8011-g6939ea03-1focal libcephfs-
    dev=17.0.0-8011-g6939ea03-1focal librados2=17.0.0-8011-g6939ea03-1focal
    librbd1=17.0.0-8011-g6939ea03-1focal rbd-fuse=17.0.0-8011-g6939ea03-1focal'

[6415947]  rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream}}
-----------------------------------------------------------------
time:   00:48:37
info:   http://pulpito.ceph.com/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/6415947/
log:    http://qa-proxy.ceph.com/teuthology/trociny-2021-10-01_06:28:15-rados-master-distro-basic-smithi/6415947/
sentry: https://sentry.ceph.com/organizations/ceph/?query=95e9568cd8f042938f08f48f2d1cdfac

    Command failed on smithi031 with status 6: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph tell osd.3 flush_pg_stats'