2020-10-19T04:31:37.549 INFO:root:teuthology version: 1.0.0-b106d09 2020-10-19T04:31:37.551 INFO:teuthology.results:subset: '913/9999' 2020-10-19T04:31:37.551 INFO:teuthology.results:seed: '2477' 2020-10-19T04:47:47.021 INFO:root:teuthology version: 1.0.0-b106d09 2020-10-19T04:47:47.079 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-10-19T05:48:48.149 INFO:teuthology.results:Tests finished! gathering results... 2020-10-19T05:48:49.464 INFO:scrape:Found 11 jobs 2020-10-19T05:48:49.466 INFO:scrape:Found 2 distinct failure reasons 2020-10-19T05:48:49.467 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi036 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=247ebb2f933580ffb084e8c7f75d673b6ca5d343 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2020-10-19T05:48:49.467 INFO:scrape:1 jobs: ['5537461'] 2020-10-19T05:48:49.467 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}'] 2020-10-19T05:48:49.467 INFO:scrape: 2020-10-19T05:48:49.468 INFO:scrape:Failure: "2020-10-19T05:09:24.684459+0000 osd.3 (osd.3) 151 : cluster [ERR] scrub 57.3 57:c6a49075:test-rados-api-smithi008-31413-74::foo:24 : size 0 != clone_size 10" in cluster log 2020-10-19T05:48:49.468 INFO:scrape:1 jobs: ['5537470'] 2020-10-19T05:48:49.468 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2020-10-19T05:48:49.469 INFO:scrape: 2020-10-19T05:48:49.469 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [5537461] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:22:19 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537461/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537461/ sentry: https://sentry.ceph.com/sentry/sepia/?query=61d3c9aee3454353905178fbd4435768 Command failed (workunit test rados/test.sh) on smithi036 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=247ebb2f933580ffb084e8c7f75d673b6ca5d343 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5537470] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:22:22 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537470/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537470/ "2020-10-19T05:09:24.684459+0000 osd.3 (osd.3) 151 : cluster [ERR] scrub 57.3 57:c6a49075:test-rados-api-smithi008-31413-74::foo:24 : size 0 != clone_size 10" in cluster log Pass ================================================================= [5537460] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/prometheus} time: 00:08:12 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537460/ [5537462] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/rados_api_tests} time: 00:35:07 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537462/ [5537463] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:19 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537463/ [5537464] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:17:14 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537464/ [5537465] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:08 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537465/ [5537466] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:19:04 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537466/ [5537467] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:32:05 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537467/ [5537468] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:28:34 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537468/ [5537469] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:05:50 info: http://pulpito.ceph.com/kchai-2020-10-19_04:31:12-rados-wip-kefu-testing-2020-10-18-2239-distro-basic-smithi/5537469/