2021-02-03T12:20:45.274 INFO:root:teuthology version: 1.0.0-c383a14 2021-02-03T12:20:45.276 INFO:teuthology.results:subset: '978/9999' 2021-02-03T12:20:45.277 INFO:teuthology.results:seed: '2477' 2021-02-03T12:26:30.658 INFO:root:teuthology version: 1.0.0-c383a14 2021-02-03T12:26:30.688 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-02-03T13:10:31.523 INFO:teuthology.results:Tests finished! gathering results... 2021-02-03T13:10:32.187 INFO:scrape:Found 14 jobs 2021-02-03T13:10:34.008 INFO:scrape:Found 3 distinct failure reasons 2021-02-03T13:10:34.009 INFO:scrape:Failure: Test failure: test_crash_history (tasks.mgr.test_insights.TestInsights) 2021-02-03T13:10:34.010 INFO:scrape:2 jobs: ['5853912', '5853922'] 2021-02-03T13:10:34.011 INFO:scrape:suites intersection: ['debug/mgr', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_8}', 'tasks/insights}'] 2021-02-03T13:10:34.012 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_8}', 'tasks/insights}'] 2021-02-03T13:10:34.013 INFO:scrape: 2021-02-03T13:10:34.014 INFO:scrape:Failure: Command failed on smithi068 with status 1: 'install -d -m0755 -- /home/ubuntu/cephtest/ceph.data' 2021-02-03T13:10:34.015 INFO:scrape:1 jobs: ['5853915'] 2021-02-03T13:10:34.015 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb', 'supported-random-distro$/{rhel_8}}'] 2021-02-03T13:10:34.016 INFO:scrape: 2021-02-03T13:10:34.017 INFO:scrape:Dead: b'2021-02-03T13:07:47.163 DEBUG:teuthology.exit:Finished running handlers' 2021-02-03T13:10:34.018 INFO:scrape:9 jobs: ['5853918', '5853916', '5853913', '5853914', '5853910', '5853917', '5853911', '5853919', '5853921'] 2021-02-03T13:10:34.020 INFO:scrape:suites intersection: [] 2021-02-03T13:10:34.021 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-tasks', '2-recovery-overrides/{more-async-recovery}', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'distro$/{ubuntu_20.04}', 'distro1$/{centos_8.0}', 'fixed-2', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'openstack', 'openstack}', 'rados', 'rados/cephadm/upgrade/{1-start', 'rados/cephadm/with-work/{distro/ubuntu_20.04', 'rados/perf/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'scheduler/dmclock_default_shards', 'scheduler/wpq_default_shards', 'settings/optimized', 'start', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests}', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'ubuntu_latest', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cosbench_64K_read_write}', 'workloads/cosbench_64K_write}', 'workloads/radosbench-high-concurrency}'] 2021-02-03T13:10:34.022 INFO:scrape: 2021-02-03T13:10:34.023 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/ failed: 3 dead: 9 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [5853912] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/insights} ----------------------------------------------------------------- time: 00:11:13 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853912/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853912/ sentry: https://sentry.ceph.com/organizations/ceph/?query=184c3e1ffa0a49db89df0df91be66a66 Test failure: test_crash_history (tasks.mgr.test_insights.TestInsights) [5853915] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:08:48 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853915/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853915/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a24991b18954bd8a84ef129f9c80ba6 Command failed on smithi068 with status 1: 'install -d -m0755 -- /home/ubuntu/cephtest/ceph.data' [5853922] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/insights} ----------------------------------------------------------------- time: 00:11:16 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853922/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853922/ sentry: https://sentry.ceph.com/organizations/ceph/?query=32f90bd134da49ebac5075623d096c99 Test failure: test_crash_history (tasks.mgr.test_insights.TestInsights) Dead ================================================================= [5853910] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{centos_8.0} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853910/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853910/ [5853911] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853911/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853911/ [5853913] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_20.04} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853913/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853913/ [5853914] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853914/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853914/ [5853916] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853916/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853916/ [5853917] rados/cephadm/with-work/{distro/ubuntu_20.04 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853917/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853917/ [5853918] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{centos_8.0} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853918/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853918/ [5853919] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853919/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853919/ [5853921] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_20.04} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853921/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853921/ Pass ================================================================= [5853909] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:09:22 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853909/ [5853920] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:09:24 info: http://pulpito.ceph.com/kchai-2021-02-03_12:18:37-rados-wip-kefu-testing-2021-02-03-1720-distro-basic-smithi/5853920/