2020-11-05T10:26:00.449 INFO:root:teuthology version: 1.0.0-41b569f 2020-11-05T10:26:00.450 INFO:teuthology.results:subset: '1385/7000' 2020-11-05T10:26:00.451 INFO:teuthology.results:seed: '4945' 2020-11-05T10:28:29.547 INFO:root:teuthology version: 1.0.0-41b569f 2020-11-05T10:28:29.552 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-11-05T22:28:42.448 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2020-11-05T22:28:45.264 INFO:scrape:Found 13 jobs 2020-11-05T22:28:46.553 INFO:scrape:Found 4 distinct failure reasons 2020-11-05T22:28:46.554 INFO:scrape:Dead: b'2020-11-05T22:26:17.174 DEBUG:teuthology.exit:Finished running handlers' 2020-11-05T22:28:46.554 INFO:scrape:1 jobs: ['5592095'] 2020-11-05T22:28:46.555 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}'] 2020-11-05T22:28:46.555 INFO:scrape: 2020-11-05T22:28:46.556 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcf4fd271c18bb40990262aa80b79e7e03ed7ad0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2020-11-05T22:28:46.556 INFO:scrape:3 jobs: ['5592100', '5592097', '5592101'] 2020-11-05T22:28:46.556 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'msgr/async', 'openstack}', 'rados'] 2020-11-05T22:28:46.557 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'thrashosds-health}', 'validater/valgrind}', 'workloads/rados_api_tests}'] 2020-11-05T22:28:46.557 INFO:scrape: 2020-11-05T22:28:46.558 INFO:scrape:Failure: ceph version 14.2.13-23-gb9795ea4-1bionic was not installed, found 14.2.13-21-g0763d1e2-1bionic. 2020-11-05T22:28:46.558 INFO:scrape:1 jobs: ['5592104'] 2020-11-05T22:28:46.559 INFO:scrape:suites: ['1-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'bluestore-bitmap', 'mon_election/connectivity', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}'] 2020-11-05T22:28:46.560 INFO:scrape: 2020-11-05T22:28:46.561 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/mon/caps.sh 2020-11-05T22:28:46.561 INFO:scrape:2 jobs: ['5592099', '5592106'] 2020-11-05T22:28:46.562 INFO:scrape:suites intersection: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'rados', 'rados/monthrash/{ceph', 'workloads/rados_mon_workunits}'] 2020-11-05T22:28:46.563 INFO:scrape:suites union: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'thrashers/force-sync-many', 'thrashers/sync', 'workloads/rados_mon_workunits}'] 2020-11-05T22:28:46.563 INFO:scrape: 2020-11-05T22:28:46.565 INFO:teuthology.results:Sending results to jfajerski@suse.com: Test Run: jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/ failed: 6 dead: 1 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [5592097] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:28:34 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592097/ log: http://qa-proxy.ceph.com/teuthology/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592097/ sentry: https://sentry.ceph.com/ceph/sepia/?query=7c0d15b558af47c0a99dcfce43c9753e Command failed (workunit test rados/test.sh) on smithi064 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcf4fd271c18bb40990262aa80b79e7e03ed7ad0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5592099] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 03:14:16 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592099/ log: http://qa-proxy.ceph.com/teuthology/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592099/ sentry: https://sentry.ceph.com/ceph/sepia/?query=e7c5fe777bd74dcdae7b3e9ee2e07667 Command failed (workunit test mon/caps.sh) on smithi100 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcf4fd271c18bb40990262aa80b79e7e03ed7ad0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh' [5592100] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:14:16 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592100/ log: http://qa-proxy.ceph.com/teuthology/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592100/ sentry: https://sentry.ceph.com/ceph/sepia/?query=9cb8e98392b24089af2bfcc3ed8943b6 Command failed (workunit test rados/test.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcf4fd271c18bb40990262aa80b79e7e03ed7ad0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5592101] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:24:41 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592101/ log: http://qa-proxy.ceph.com/teuthology/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592101/ sentry: https://sentry.ceph.com/ceph/sepia/?query=1fb5bed75233451d8b217409dd34b6f9 Command failed (workunit test rados/test.sh) on smithi001 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcf4fd271c18bb40990262aa80b79e7e03ed7ad0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5592104] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:04:19 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592104/ log: http://qa-proxy.ceph.com/teuthology/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592104/ sentry: https://sentry.ceph.com/ceph/sepia/?query=8a7a45fccd84487691923ca726a4a836 ceph version 14.2.13-23-gb9795ea4-1bionic was not installed, found 14.2.13-21-g0763d1e2-1bionic. [5592106] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 03:19:18 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592106/ log: http://qa-proxy.ceph.com/teuthology/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592106/ sentry: https://sentry.ceph.com/ceph/sepia/?query=4ceff1712f1946fea32ef08a8c79d2b2 Command failed (workunit test mon/caps.sh) on smithi073 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fcf4fd271c18bb40990262aa80b79e7e03ed7ad0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh' Dead ================================================================= [5592095] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592095/ log: http://qa-proxy.ceph.com/teuthology/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592095/ Pass ================================================================= [5592096] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:15:31 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592096/ [5592098] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:20:14 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592098/ [5592102] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:12:08 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592102/ [5592103] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:25:39 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592103/ [5592105] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects} time: 00:18:08 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592105/ [5592107] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:28:38 info: http://pulpito.ceph.com/jafaj-2020-11-05_07:50:23-rados-wip-jan-testing-2020-10-29-1114-distro-basic-smithi/5592107/