2021-03-28T03:14:49.710 INFO:root:teuthology version: 1.1.0-d5a60f86 2021-03-28T03:14:49.710 INFO:teuthology.results:subset: '10023/99999' 2021-03-28T03:14:49.710 INFO:teuthology.results:seed: '2477' 2021-03-28T04:31:43.755 INFO:root:teuthology version: 1.1.0-d5a60f86 2021-03-28T04:31:43.789 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-28T11:59:33.116 INFO:teuthology.results:Tests finished! gathering results... 2021-03-28T11:59:35.290 INFO:scrape:Found 18 jobs 2021-03-28T11:59:36.549 INFO:scrape:Found 3 distinct failure reasons 2021-03-28T11:59:36.550 INFO:scrape:Failure: Command failed on smithi106 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3e31def9543c902ee0dbe0a9dd079680aaa5a200 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 834a2208-8f7c-11eb-ab34-001a4aab830c -- ceph orch host ls --format=json' 2021-03-28T11:59:36.550 INFO:scrape:1 jobs: ['6005060'] 2021-03-28T11:59:36.550 INFO:scrape:suites: ['fixed-2', 'mon_election/classic', 'rados/cephadm/smoke/{distro/ubuntu_20.04_kubic_stable', 'start}'] 2021-03-28T11:59:36.550 INFO:scrape: 2021-03-28T11:59:36.550 INFO:scrape:Failure: Exiting scrub checking -- not all pgs scrubbed. 2021-03-28T11:59:36.550 INFO:scrape:1 jobs: ['6005054'] 2021-03-28T11:59:36.551 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache-snaps-balanced}'] 2021-03-28T11:59:36.551 INFO:scrape: 2021-03-28T11:59:36.551 INFO:scrape:Dead: b'2021-03-28T11:58:12.995 DEBUG:teuthology.exit:Finished running handlers' 2021-03-28T11:59:36.551 INFO:scrape:1 jobs: ['6005055'] 2021-03-28T11:59:36.551 INFO:scrape:suites: ['1-tasks', 'distro1$/{ubuntu_20.04}', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-03-28T11:59:36.551 INFO:scrape: 2021-03-28T11:59:36.552 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/ failed: 2 dead: 1 running: 0 waiting: 0 queued: 0 passed: 15 Fail ================================================================= [6005054] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 01:04:47 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005054/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005054/ Exiting scrub checking -- not all pgs scrubbed. [6005060] rados/cephadm/smoke/{distro/ubuntu_20.04_kubic_stable fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:06:42 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005060/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005060/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1fbf6d2cdcec468fa5a1a2e7491e2a0e Command failed on smithi106 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:3e31def9543c902ee0dbe0a9dd079680aaa5a200 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 834a2208-8f7c-11eb-ab34-001a4aab830c -- ceph orch host ls --format=json' Dead ================================================================= [6005055] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_20.04} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005055/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005055/ Pass ================================================================= [6005056] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/valgrind} time: 03:25:18 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005056/ [6005057] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04_kubic_stable 1-start 2-services/mirror 3-final} time: 00:15:59 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005057/ [6005058] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:30:39 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005058/ [6005059] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:01 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005059/ [6005061] rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04_kubic_testing 1-start 2-services/rgw 3-final} time: 00:16:21 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005061/ [6005062] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:18:03 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005062/ [6005063] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:12:28 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005063/ [6005064] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:34:15 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005064/ [6005065] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/mon_recovery validater/valgrind} time: 00:28:44 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005065/ [6005066] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_8} tasks/dashboard} time: 00:55:28 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005066/ [6005067] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_20.04_kubic_testing} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:28:50 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005067/ [6005068] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:16 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005068/ [6005069] rados/cephadm/smoke/{distro/ubuntu_20.04_kubic_stable fixed-2 mon_election/classic start} time: 00:13:48 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005069/ [6005070] rados/upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} time: 00:25:58 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005070/ [6005071] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:35:14 info: http://pulpito.ceph.com/kchai-2021-03-28_03:01:25-rados-wip-kefu-testing-2021-03-27-1545-distro-basic-smithi/6005071/