2021-02-03T14:19:28.270 INFO:root:teuthology version: 1.0.0-c383a14 2021-02-03T14:19:28.273 INFO:teuthology.results:subset: '978/9999' 2021-02-03T14:19:28.273 INFO:teuthology.results:seed: '2477' 2021-02-03T14:36:55.833 INFO:root:teuthology version: 1.0.0-c383a14 2021-02-03T14:36:55.856 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-02-04T02:37:09.328 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-02-04T02:37:10.934 INFO:scrape:Found 17 jobs 2021-02-04T02:37:13.423 INFO:scrape:Found 8 distinct failure reasons 2021-02-04T02:37:13.424 INFO:scrape:Dead: b'2021-02-03T15:01:51.909 INFO:journalctl@ceph.mon.c.smithi035.stdout:Feb 03 15:01:51 smithi035 bash[15704]: audit 2021-02-03T15:01:51.367945+0000 mon.a (mon.0) 1572 : audit [INF] from=\'client.37358 \' entity=\'client.admin\' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi035-34369-29","app": "rados","yes_i_really_mean_it": true}]: dispatch' 2021-02-04T02:37:13.424 INFO:scrape:1 jobs: ['5853959'] 2021-02-04T02:37:13.425 INFO:scrape:suites: ['fixed-2', 'mode/root', 'mon_election/classic', 'msgr/async-v1only', 'rados/cephadm/with-work/{distro/ubuntu_20.04', 'start', 'tasks/rados_api_tests}'] 2021-02-04T02:37:13.425 INFO:scrape: 2021-02-04T02:37:13.426 INFO:scrape:Failure: Command failed on smithi166 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 007415e4-662f-11eb-8fbe-001a4aab830c -e sha1=c72b89d7e84e5cb38b1b895a7ca6a3afec7e52ad -- bash -c 'ceph orch ps'" 2021-02-04T02:37:13.426 INFO:scrape:1 jobs: ['5853960'] 2021-02-04T02:37:13.427 INFO:scrape:suites: ['1-tasks', 'distro1$/{centos_8.0}', 'mon_election/classic', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-02-04T02:37:13.427 INFO:scrape: 2021-02-04T02:37:13.428 INFO:scrape:Dead: b'2021-02-03T14:47:15.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored' 2021-02-04T02:37:13.428 INFO:scrape:1 jobs: ['5853948'] 2021-02-04T02:37:13.428 INFO:scrape:suites: ['mon_election/classic', 'objectstore/bluestore-stupid', 'openstack', 'rados/perf/{ceph', 'scheduler/dmclock_default_shards', 'settings/optimized', 'ubuntu_latest', 'workloads/radosbench_omap_write}'] 2021-02-04T02:37:13.429 INFO:scrape: 2021-02-04T02:37:13.429 INFO:scrape:Failure: Command failed on smithi142 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac129314-662d-11eb-8fbe-001a4aab830c -e sha1=c72b89d7e84e5cb38b1b895a7ca6a3afec7e52ad -- bash -c 'ceph orch ps'" 2021-02-04T02:37:13.430 INFO:scrape:1 jobs: ['5853951'] 2021-02-04T02:37:13.430 INFO:scrape:suites: ['1-tasks', 'distro1$/{centos_8.0}', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-02-04T02:37:13.431 INFO:scrape: 2021-02-04T02:37:13.431 INFO:scrape:Dead: b'2021-02-03T14:36:39.333 INFO:teuthology.orchestra.run.smithi008.stdout:Booting cosbench driver ...' 2021-02-04T02:37:13.432 INFO:scrape:2 jobs: ['5853953', '5853956'] 2021-02-04T02:37:13.432 INFO:scrape:suites intersection: ['openstack', 'rados/perf/{ceph', 'settings/optimized', 'ubuntu_latest'] 2021-02-04T02:37:13.432 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/dmclock_default_shards', 'scheduler/wpq_default_shards', 'settings/optimized', 'ubuntu_latest', 'workloads/cosbench_64K_read_write}', 'workloads/cosbench_64K_write}'] 2021-02-04T02:37:13.433 INFO:scrape: 2021-02-04T02:37:13.433 INFO:scrape:Failure: SSH connection to smithi014 was lost: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bff7ee38-662d-11eb-8fbe-001a4aab830c --force' 2021-02-04T02:37:13.434 INFO:scrape:1 jobs: ['5853955'] 2021-02-04T02:37:13.434 INFO:scrape:suites: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{ubuntu_20.04}', 'fixed-2', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2021-02-04T02:37:13.434 INFO:scrape: 2021-02-04T02:37:13.435 INFO:scrape:Dead: b'2021-02-03T15:03:55.568 INFO:journalctl@ceph.alertmanager.a.smithi031.stdout:Feb 03 15:03:55 smithi031 bash[25054]: level=info ts=2021-02-03T15:03:55.165Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..."' 2021-02-04T02:37:13.435 INFO:scrape:1 jobs: ['5853963'] 2021-02-04T02:37:13.436 INFO:scrape:suites: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{ubuntu_20.04}', 'fixed-2', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2021-02-04T02:37:13.436 INFO:scrape: 2021-02-04T02:37:13.437 INFO:scrape:Dead: b'2021-02-03T14:50:15.643 INFO:journalctl@ceph.mon.c.smithi149.stdout:Feb 03 14:50:15 smithi149 bash[15626]: cluster 2021-02-03T14:50:15.244613+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in' 2021-02-04T02:37:13.437 INFO:scrape:1 jobs: ['5853950'] 2021-02-04T02:37:13.437 INFO:scrape:suites: ['fixed-2', 'mode/packaged', 'mon_election/connectivity', 'msgr/async-v2only', 'rados/cephadm/with-work/{distro/ubuntu_20.04', 'start', 'tasks/rados_python}'] 2021-02-04T02:37:13.438 INFO:scrape: 2021-02-04T02:37:13.439 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/ failed: 2 dead: 1 running: 6 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [5853951] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{centos_8.0} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:58:34 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853951/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853951/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54a52f1677a74b6693755d44322d1b3b Command failed on smithi142 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac129314-662d-11eb-8fbe-001a4aab830c -e sha1=c72b89d7e84e5cb38b1b895a7ca6a3afec7e52ad -- bash -c 'ceph orch ps'" [5853960] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{centos_8.0} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:58:15 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853960/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853960/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8b7d229e1f745268b92557d1f36f953 Command failed on smithi166 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 007415e4-662f-11eb-8fbe-001a4aab830c -e sha1=c72b89d7e84e5cb38b1b895a7ca6a3afec7e52ad -- bash -c 'ceph orch ps'" Dead ================================================================= [5853955] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_20.04} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:50:47 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853955/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853955/ SSH connection to smithi014 was lost: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bff7ee38-662d-11eb-8fbe-001a4aab830c --force' Running ================================================================= [5853948] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853948/ [5853950] rados/cephadm/with-work/{distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python} info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853950/ [5853953] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853953/ [5853956] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853956/ [5853959] rados/cephadm/with-work/{distro/ubuntu_20.04 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests} info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853959/ [5853963] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_20.04} fixed-2 mon_election/connectivity} info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853963/ Pass ================================================================= [5853949] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:09:31 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853949/ [5853952] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:36:23 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853952/ [5853954] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/insights} time: 00:15:17 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853954/ [5853957] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:18:01 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853957/ [5853958] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:52:06 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853958/ [5853961] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:25:30 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853961/ [5853962] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:09:25 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853962/ [5853964] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/insights} time: 00:15:11 info: http://pulpito.ceph.com/kchai-2021-02-03_14:17:43-rados-wip-kefu-testing-2021-02-03-2051-distro-basic-smithi/5853964/