2021-01-20T02:04:54.715 INFO:root:teuthology version: 1.0.0-0746baa9 2021-01-20T02:04:54.717 INFO:teuthology.results:subset: '1/2' 2021-01-20T02:04:54.717 INFO:teuthology.results:seed: '2527' 2021-01-20T02:38:58.224 INFO:root:teuthology version: 1.0.0-0746baa9 2021-01-20T02:38:58.274 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-20T03:02:58.817 INFO:teuthology.results:Tests finished! gathering results... 2021-01-20T03:03:21.400 INFO:scrape:Found 74 jobs 2021-01-20T03:03:22.574 INFO:scrape:Found 3 distinct failure reasons 2021-01-20T03:03:22.575 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2021-01-20T03:03:22.575 INFO:scrape:65 jobs 2021-01-20T03:03:22.576 INFO:scrape:suites intersection: [] 2021-01-20T03:03:22.576 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '2-kms/testing', '2-kms/vault_kv', '2-kms/vault_transit', '3-rgw/rgw', '4-tests/{s3tests}}', 'cls', 'cluster', 'clusters/fixed-2', 'frontend/beast', 'frontend/civetweb', 'hadoop/default', 'hadoop/v32', 'http', 'install', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/hadoop-s3a/{clusters/fixed-2', 'rgw/multifs/{clusters/fixed-2', 'rgw/singleton/{all/radosgw-admin', 'rgw/sts/{centos_latest', 'rgw/tempest/{clusters/fixed-1', 'rgw/thrash/{civetweb', 'rgw/tools/{centos_latest', 'rgw/verify/{centos_latest', 'rgw/website/{clusters/fixed-2', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3a-hadoop}', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'ststests', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/rgw_bucket_quota}', 'tasks/rgw_multipart_upload}', 'tasks/rgw_ragweed}', 'tasks/rgw_s3tests}', 'tasks/rgw_tempest', 'tasks/rgw_user_quota}', 'tasks/s3tests-website', 'tasks/{0-install', 'tasks}', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest}', 'validater/lockdep}', 'validater/valgrind}', 'webidentity}}', 'workload/rgw_bucket_quota}', 'workload/rgw_multipart_upload}', 'workload/rgw_s3tests}', 'workload/rgw_user_quota}'] 2021-01-20T03:03:22.577 INFO:scrape: 2021-01-20T03:03:22.577 INFO:scrape:Crash: "2021-01-20T02:44:50.272423+0000 mon.a (mon.0) 137 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log 2021-01-20T03:03:22.578 INFO:scrape:ceph version 16.0.0-9150-gba66e523 (ba66e523ea8b03638d362f61d5a6bef9fd1f825f) pacific (dev) 1: /lib64/libpthread.so.0(+0x12dc0) [0x7f85d4b89dc0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f85d5f9da6b] 5: /usr/lib64/ceph/libceph-common.so.2(+0x275c34) [0x7f85d5f9dc34] 6: (Mgr::load_all_metadata()+0x1ecd) [0x5609bd45946d] 7: (Mgr::init()+0x6cc) [0x5609bd459d5c] 8: ceph-mgr(+0x226bc1) [0x5609bd45abc1] 9: (Context::complete(int)+0xd) [0x5609bd3c0bdd] 10: (Finisher::finisher_thread_entry()+0x1a5) [0x7f85d603c5d5] 11: /lib64/libpthread.so.0(+0x82de) [0x7f85d4b7f2de] 12: clone() 2021-01-20T03:03:22.578 INFO:scrape:8 jobs: ['5800440', '5800418', '5800445', '5800472', '5800428', '5800415', '5800420', '5800465'] 2021-01-20T03:03:22.579 INFO:scrape:suites intersection: ['overrides'] 2021-01-20T03:03:22.579 INFO:scrape:suites union: ['cls', 'clusters/fixed-2', 'frontend/beast', 'frontend/civetweb', 'http', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'omap_limits', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'reshard', 'rgw/multisite/{clusters', 'rgw/verify/{centos_latest', 'rgw/website/{clusters/fixed-2', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/s3tests-website', 'tasks/test_multi', 'tasks/{0-install', 'ubuntu_latest}', 'valgrind}', 'validater/lockdep}', 'validater/valgrind}'] 2021-01-20T03:03:22.580 INFO:scrape: 2021-01-20T03:03:22.580 INFO:scrape:Timeout 120 running ceph --cluster c1 pg dump --format=json 2021-01-20T03:03:22.580 INFO:scrape:1 jobs: ['5800408'] 2021-01-20T03:03:22.581 INFO:scrape:suites: ['frontend/beast', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2021-01-20T03:03:22.581 INFO:scrape: