2020-12-05T08:44:12.460 INFO:root:teuthology version: 1.0.0-e6b33b7b 2020-12-05T08:44:12.461 INFO:teuthology.results:subset: '937/9999' 2020-12-05T08:44:12.461 INFO:teuthology.results:seed: '2477' 2020-12-05T08:52:08.751 INFO:root:teuthology version: 1.0.0-e6b33b7b 2020-12-05T08:52:08.771 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-05T20:52:21.948 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2020-12-05T20:52:23.281 INFO:scrape:Found 15 jobs 2020-12-05T20:52:25.018 INFO:scrape:Found 7 distinct failure reasons 2020-12-05T20:52:25.019 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-12-05T20:52:25.020 INFO:scrape:3 jobs: ['5682574', '5682578', '5682581'] 2020-12-05T20:52:25.020 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] 2020-12-05T20:52:25.021 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-12-05T20:52:25.021 INFO:scrape: 2020-12-05T20:52:25.021 INFO:scrape:Dead: b'2020-12-05T20:50:28.467 DEBUG:teuthology.exit:Finished running handlers' 2020-12-05T20:52:25.022 INFO:scrape:1 jobs: ['5682587'] 2020-12-05T20:52:25.022 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}'] 2020-12-05T20:52:25.023 INFO:scrape: 2020-12-05T20:52:25.023 INFO:scrape:Failure: psutil.NoSuchProcess process no longer exists (pid=22319) 2020-12-05T20:52:25.023 INFO:scrape:1 jobs: ['5682579'] 2020-12-05T20:52:25.024 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}'] 2020-12-05T20:52:25.024 INFO:scrape: 2020-12-05T20:52:25.025 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0c181d070c46e8266b4307971bc43b2cf1aa276d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2020-12-05T20:52:25.025 INFO:scrape:2 jobs: ['5682583', '5682577'] 2020-12-05T20:52:25.026 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2020-12-05T20:52:25.026 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2020-12-05T20:52:25.026 INFO:scrape: 2020-12-05T20:52:25.027 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi186 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0c181d070c46e8266b4307971bc43b2cf1aa276d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2020-12-05T20:52:25.027 INFO:scrape:2 jobs: ['5682586', '5682580'] 2020-12-05T20:52:25.028 INFO:scrape:suites intersection: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] 2020-12-05T20:52:25.028 INFO:scrape:suites union: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] 2020-12-05T20:52:25.028 INFO:scrape: 2020-12-05T20:52:25.029 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 2020-12-05T20:52:25.029 INFO:scrape:1 jobs: ['5682575'] 2020-12-05T20:52:25.029 INFO:scrape:suites: ['mon_election/classic', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_orch_cli}'] 2020-12-05T20:52:25.030 INFO:scrape: 2020-12-05T20:52:25.030 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2020-12-05T20:52:25.031 INFO:scrape:1 jobs: ['5682582'] 2020-12-05T20:52:25.031 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}'] 2020-12-05T20:52:25.031 INFO:scrape: 2020-12-05T20:52:25.032 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/ failed: 8 dead: 2 running: 1 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [5682574] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:18:25 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682574/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682574/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dfec06cf64714f52992f4bbdc212a917 reached maximum tries (180) after waiting for 180 seconds [5682575] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:10:52 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682575/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682575/ sentry: https://sentry.ceph.com/organizations/ceph/?query=924b653df85d457fa33efacae156a4b2 Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) [5682577] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:07:21 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682577/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682577/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e8386337ac14fb383510ae88261e72c Command failed (workunit test cephadm/test_adoption.sh) on smithi083 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0c181d070c46e8266b4307971bc43b2cf1aa276d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5682578] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:14:54 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682578/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682578/ sentry: https://sentry.ceph.com/organizations/ceph/?query=05d734118f3e4c8a98c4813b74f79aec reached maximum tries (180) after waiting for 180 seconds [5682580] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} ----------------------------------------------------------------- time: 00:14:21 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682580/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682580/ sentry: https://sentry.ceph.com/organizations/ceph/?query=17f19510c72f403eaaf4a88f90055cbb Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi131 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0c181d070c46e8266b4307971bc43b2cf1aa276d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [5682581] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:18:46 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682581/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682581/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd2e0892faab409da0b738582b81923c reached maximum tries (180) after waiting for 180 seconds [5682583] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_adoption} ----------------------------------------------------------------- time: 00:07:15 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682583/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682583/ sentry: https://sentry.ceph.com/organizations/ceph/?query=146cb730380e405aa4249bd0825997c1 Command failed (workunit test cephadm/test_adoption.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0c181d070c46e8266b4307971bc43b2cf1aa276d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5682586] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} ----------------------------------------------------------------- time: 00:14:30 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682586/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682586/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db4eeb8b56a84932a6390f04438dfee4 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi186 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0c181d070c46e8266b4307971bc43b2cf1aa276d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' Dead ================================================================= [5682579] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 11:52:05 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682579/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682579/ psutil.NoSuchProcess process no longer exists (pid=22319) [5682582] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 11:53:02 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682582/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682582/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f26ed26024f4ff5b729f5a3da0b04e4 Command failed (workunit test rados/test.sh) on smithi195 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0c181d070c46e8266b4307971bc43b2cf1aa276d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' Running ================================================================= [5682587] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_cls_all validater/valgrind} info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682587/ Pass ================================================================= [5682573] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_cls_all validater/valgrind} time: 03:27:40 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682573/ [5682576] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/valgrind} time: 00:34:40 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682576/ [5682584] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/valgrind} time: 00:28:17 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682584/ [5682585] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:28:53 info: http://pulpito.ceph.com/kchai-2020-12-05_08:42:04-rados-wip-kefu-testing-2020-12-05-1217-distro-basic-smithi/5682585/