2021-03-08T10:56:47.722 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-08T10:56:47.728 INFO:teuthology.results:subset: '989/99999' 2021-03-08T10:56:47.729 INFO:teuthology.results:seed: '2477' 2021-03-08T11:05:47.595 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-08T11:05:47.622 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-08T13:00:56.088 INFO:teuthology.results:Tests finished! gathering results... 2021-03-08T13:00:57.256 INFO:scrape:Found 10 jobs 2021-03-08T13:00:58.921 INFO:scrape:Found 3 distinct failure reasons 2021-03-08T13:00:58.922 INFO:scrape:Failure: rgw multisite test failures 2021-03-08T13:00:58.923 INFO:scrape:1 jobs: ['5946443'] 2021-03-08T13:00:58.924 INFO:scrape:suites: ['frontend', 'overrides', 'rados/upgrade/pacific-x/rgw-multisite/{clusters', 'realm', 'tasks', 'upgrade/primary}'] 2021-03-08T13:00:58.924 INFO:scrape: 2021-03-08T13:00:58.925 INFO:scrape:Failure: Command failed on smithi087 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fc09ffc-7fff-11eb-9078-001a4aab830c -e sha1=90746a7324b8dcb99d01fb389c8693a7978e8d61 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2021-03-08T13:00:58.925 INFO:scrape:2 jobs: ['5946447', '5946450'] 2021-03-08T13:00:58.926 INFO:scrape:suites intersection: ['1-tasks', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-03-08T13:00:58.926 INFO:scrape:suites union: ['1-tasks', 'distro1$/{ubuntu_20.04_kubic_testing}', 'distro1$/{ubuntu_20.04}', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-03-08T13:00:58.927 INFO:scrape: 2021-03-08T13:00:58.927 INFO:scrape:Failure: Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=90746a7324b8dcb99d01fb389c8693a7978e8d61 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' 2021-03-08T13:00:58.928 INFO:scrape:4 jobs: ['5946448', '5946445', '5946449', '5946451'] 2021-03-08T13:00:58.929 INFO:scrape:suites intersection: ['task/test_e2e}'] 2021-03-08T13:00:58.929 INFO:scrape:suites union: ['rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable', 'rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing', 'task/test_e2e}'] 2021-03-08T13:00:58.930 INFO:scrape: 2021-03-08T13:00:58.931 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 3 Fail ================================================================= [5946443] rados/upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:26:34 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946443/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946443/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b789a1d2b4b14283b3e7d35196bab7e0 rgw multisite test failures [5946445] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing task/test_e2e} ----------------------------------------------------------------- time: 00:13:27 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946445/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946445/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7d93200747c14f459341a8928df65db8 Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi175 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=90746a7324b8dcb99d01fb389c8693a7978e8d61 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' [5946447] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_20.04_kubic_testing} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:17:17 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946447/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946447/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f553eab1b0c14c85bc3eb168060e4434 Command failed on smithi087 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fc09ffc-7fff-11eb-9078-001a4aab830c -e sha1=90746a7324b8dcb99d01fb389c8693a7978e8d61 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [5946448] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable task/test_e2e} ----------------------------------------------------------------- time: 00:14:39 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946448/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946448/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7099255df2343a39fc0e0ddbe554dc2 Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=90746a7324b8dcb99d01fb389c8693a7978e8d61 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' [5946449] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing task/test_e2e} ----------------------------------------------------------------- time: 00:14:31 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946449/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946449/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aaaf624124994caca7c72bb5be6d653d Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=90746a7324b8dcb99d01fb389c8693a7978e8d61 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' [5946450] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_20.04} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:17:09 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946450/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946450/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f832e8e150304233924dea4f67105827 Command failed on smithi132 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa1eed3c-7fff-11eb-9078-001a4aab830c -e sha1=90746a7324b8dcb99d01fb389c8693a7978e8d61 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [5946451] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable task/test_e2e} ----------------------------------------------------------------- time: 00:14:43 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946451/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946451/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d42df4790594732bc9231dedfd552ff Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=90746a7324b8dcb99d01fb389c8693a7978e8d61 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' Pass ================================================================= [5946442] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:17 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946442/ [5946444] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench} time: 01:51:46 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946444/ [5946446] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:48 info: http://pulpito.ceph.com/kchai-2021-03-08_10:48:51-rados-wip-kefu2-testing-2021-03-08-1335-distro-basic-smithi/5946446/