2020-12-20T12:16:23.487 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-20T12:16:23.489 INFO:teuthology.results:subset: '945/9999' 2020-12-20T12:16:23.489 INFO:teuthology.results:seed: '2477' 2020-12-20T12:30:45.187 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-20T12:30:45.192 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-21T00:30:57.382 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2020-12-21T00:30:58.690 INFO:scrape:Found 18 jobs 2020-12-21T00:31:00.894 INFO:scrape:Found 3 distinct failure reasons 2020-12-21T00:31:00.895 INFO:scrape:Dead: b'2020-12-21T00:16:38.941 DEBUG:teuthology.exit:Finished running handlers' 2020-12-21T00:31:00.895 INFO:scrape:8 jobs: ['5723461', '5723474', '5723476', '5723471', '5723473', '5723472', '5723468', '5723464'] 2020-12-21T00:31:00.895 INFO:scrape:suites intersection: ['openstack}', 'rados'] 2020-12-21T00:31:00.896 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'd-thrash/default/{default', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/verify/{centos_latest', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'thrashers/mapgap', 'thrashosds-health', 'thrashosds-health}', 'validater/valgrind}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/small-objects-balanced}'] 2020-12-21T00:31:00.896 INFO:scrape: 2020-12-21T00:31:00.897 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-12-21T00:31:00.897 INFO:scrape:6 jobs: ['5723463', '5723465', '5723475', '5723470', '5723467', '5723462'] 2020-12-21T00:31:00.898 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] 2020-12-21T00:31:00.898 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_8.0}', 'distro$/{centos_latest}', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-12-21T00:31:00.898 INFO:scrape: 2020-12-21T00:31:00.899 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=da0ced08d3f7f2368d710e4e4e2ee7da5b7ddff3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2020-12-21T00:31:00.899 INFO:scrape:2 jobs: ['5723477', '5723469'] 2020-12-21T00:31:00.900 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-12-21T00:31:00.900 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-12-21T00:31:00.900 INFO:scrape: 2020-12-21T00:31:00.902 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/ failed: 8 dead: 7 running: 1 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [5723462] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:17:27 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723462/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca5366842a1843fab4f7957f51df45e7 reached maximum tries (180) after waiting for 180 seconds [5723463] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:17:24 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723463/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723463/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b9a7e86c3470449e91aabfe628cbcfbc reached maximum tries (180) after waiting for 180 seconds [5723465] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:16:32 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723465/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7fce6f500d744531b94dcb2231b9d8a9 reached maximum tries (180) after waiting for 180 seconds [5723467] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:17:29 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723467/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723467/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0af6c8c27ed243a692084d5a079a38a1 reached maximum tries (180) after waiting for 180 seconds [5723469] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:12:33 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723469/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723469/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7eeb6705aa844705b53d04986f615f41 Command failed (workunit test cephadm/test_cephadm.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=da0ced08d3f7f2368d710e4e4e2ee7da5b7ddff3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5723470] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:26 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723470/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723470/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc9e9f2f62a247918d131865d8d7b913 reached maximum tries (180) after waiting for 180 seconds [5723475] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:18 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723475/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723475/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51bb42c763c949f8a882805f3a25e089 reached maximum tries (180) after waiting for 180 seconds [5723477] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:07:57 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723477/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723477/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8253037b110844ab8f268174cb3b753c Command failed (workunit test cephadm/test_cephadm.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=da0ced08d3f7f2368d710e4e4e2ee7da5b7ddff3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' Dead ================================================================= [5723461] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723461/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723461/ [5723464] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723464/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723464/ [5723468] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723468/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723468/ [5723471] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723471/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723471/ [5723472] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723472/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723472/ [5723473] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723473/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723473/ [5723474] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723474/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723474/ Running ================================================================= [5723476] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/radosbench} info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723476/ Pass ================================================================= [5723460] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} time: 00:27:20 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723460/ [5723466] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:24:53 info: http://pulpito.ceph.com/kchai-2020-12-20_12:15:20-rados-wip-kefu-testing-2020-12-17-1542-distro-basic-smithi/5723466/