2020-05-17T05:41:19.774 INFO:root:teuthology version: 1.0.0-437df57 2020-05-17T05:41:19.776 INFO:teuthology.results:subset: '814/9999' 2020-05-17T05:41:19.776 INFO:teuthology.results:seed: '2477' 2020-05-17T10:27:34.292 INFO:root:teuthology version: 1.0.0-437df57 2020-05-17T10:27:34.323 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-05-17T21:55:47.729 INFO:teuthology.results:Tests finished! gathering results... 2020-05-17T21:55:47.818 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/ failed: 8 dead: 37 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [5062839] rados/singleton/{all/test_envlibrados_for_rocksdb.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-comp-lz4.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:09:49 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062839/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062839/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=b1afc9d1b8464f108b8746419858c3e6 Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi204 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2072b2cf276c3666f8002c1c445130ad526c7176 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' [5062852] rados/mgr/{clusters/{2-node-mgr.yaml} debug/mgr.yaml objectstore/bluestore-comp-zstd.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/module_selftest.yaml} ----------------------------------------------------------------- time: 00:10:06 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062852/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062852/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=3f4058660c0547988a6ca7190aaef622 Test failure: test_diskprediction_local (tasks.mgr.test_module_selftest.TestModuleSelftest) [5062856] rados/verify/{centos_latest.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/none.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml tasks/rados_cls_all.yaml validater/valgrind.yaml} ----------------------------------------------------------------- time: 01:08:41 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062856/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062856/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=a6a4d13639d64e3aa56a8fb181b15ac1 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2072b2cf276c3666f8002c1c445130ad526c7176 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [5062867] rados/objectstore/{backends/objectstore.yaml supported-random-distro$/{centos_8.yaml}} ----------------------------------------------------------------- time: 02:05:50 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062867/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062867/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=72774585829d4045a512bae522064d36 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-filestore 20 --debug- bluestore 20" ceph_test_objectstore --gtest_filter=-*/3 --gtest_catch_exceptions=0\'' [5062870] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/nautilus.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-octopus.yaml 8-workload/{rbd-python.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} ----------------------------------------------------------------- time: 00:03:24 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062870/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062870/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=4ac377d4eda146e1b98b77cce9238f0c Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref= nautilus [5062874] rados/singleton-nomsgr/{all/cache-fs-trunc.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:07:42 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062874/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062874/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=42cc28ed41e74c0daac6217bb6d0e4a2 Command failed on smithi185 with status 1: "sudo bash -c 'echo 0 > /tmp/python-ceph-brx'" [5062875] rados/verify/{centos_latest.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/none.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp-zlib.yaml rados.yaml tasks/rados_cls_all.yaml validater/valgrind.yaml} ----------------------------------------------------------------- time: 02:07:20 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062875/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062875/ saw valgrind issues [5062876] rados/cephadm/workunits/{distro/ubuntu_latest.yaml task/test_cephadm_repos.yaml} ----------------------------------------------------------------- time: 00:03:29 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062876/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062876/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=e5a1154256244ff18587bdc0f180e297 Command failed (workunit test cephadm/test_repos.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2072b2cf276c3666f8002c1c445130ad526c7176 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' Dead ================================================================= [5062837] rados/dashboard/{clusters/{2-node-mgr.yaml} debug/mgr.yaml objectstore/bluestore-comp-zstd.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/dashboard.yaml} ----------------------------------------------------------------- time: 00:37:04 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062837/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062837/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=b4040b092701474dab021485999e3184 Test failure: setUpClass (tasks.mgr.dashboard.test_host.HostControllerTest) [5062838] rados/objectstore/{backends/filestore-idempotent.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062838/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062838/ [5062840] rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-comp-lz4.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} ----------------------------------------------------------------- time: 00:36:33 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062840/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062840/ SSH connection to smithi101 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no- omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5062842] rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_workunit_loadgen_mix.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062842/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062842/ [5062843] rados/dashboard/{clusters/{2-node-mgr.yaml} debug/mgr.yaml objectstore/bluestore-hybrid.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/dashboard.yaml} ----------------------------------------------------------------- time: 00:51:46 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062843/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062843/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=802eb089f7d648309b7996f3b69569cd Test failure: setUpClass (tasks.mgr.dashboard.test_host.HostControllerTest) [5062844] rados/perf/{ceph.yaml objectstore/bluestore-comp.yaml openstack.yaml settings/optimized.yaml ubuntu_latest.yaml workloads/radosbench_omap_write.yaml} ----------------------------------------------------------------- time: 00:35:49 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062844/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062844/ unexpected end of data [5062845] rados/singleton/{all/thrash_cache_writeback_proxy_none.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-hybrid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062845/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062845/ [5062846] rados/standalone/{supported-random-distro$/{ubuntu_latest.yaml} workloads/mon.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062846/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062846/ [5062847] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/on.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/snaps-few-objects-localized.yaml} ----------------------------------------------------------------- time: 00:35:36 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062847/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062847/ SSH connection to smithi185 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5062848] rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062848/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062848/ [5062849] rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-hybrid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_workunit_loadgen_mostlyread.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062849/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062849/ [5062850] rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync-many.yaml workloads/rados_mon_osdmap_prune.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062850/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062850/ [5062851] rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-comp-zlib.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} ----------------------------------------------------------------- time: 00:36:28 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062851/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062851/ SSH connection to smithi037 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no- omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5062853] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/on.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/cache-agent-big.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062853/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062853/ [5062854] rados/perf/{ceph.yaml objectstore/bluestore-stupid.yaml openstack.yaml settings/optimized.yaml ubuntu_latest.yaml workloads/cosbench_64K_read_write.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062854/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062854/ [5062855] rados/cephadm/workunits/{distro/ubuntu_latest.yaml task/test_cephadm.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062855/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062855/ [5062857] rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} ----------------------------------------------------------------- time: 00:35:55 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062857/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062857/ SSH connection to smithi107 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no- omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5062858] rados/perf/{ceph.yaml objectstore/bluestore-basic-min-osd-mem-target.yaml openstack.yaml settings/optimized.yaml ubuntu_latest.yaml workloads/cosbench_64K_write.yaml} ----------------------------------------------------------------- time: 12:07:52 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062858/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062858/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=abafcda730714352a964e722fdf6f021 psutil.NoSuchProcess process no longer exists (pid=10042) [5062859] rados/dashboard/{clusters/{2-node-mgr.yaml} debug/mgr.yaml objectstore/bluestore-stupid.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/dashboard.yaml} ----------------------------------------------------------------- time: 00:36:04 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062859/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062859/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=f6be2cfe805a42c8b1b8fad4e56e8e28 Test failure: setUpClass (tasks.mgr.dashboard.test_host.HostControllerTest) [5062860] rados/objectstore/{backends/objectcacher-stress.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062860/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062860/ [5062861] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-partial-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/on.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062861/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062861/ [5062862] rados/cephadm/with-work/{distro/ubuntu_latest.yaml fixed-2.yaml mode/root.yaml msgr/async-v1only.yaml start.yaml tasks/rados_api_tests.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062862/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062862/ [5062863] rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-comp-lz4.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062863/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062863/ [5062864] rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=4-m=2.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062864/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062864/ [5062865] rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-comp-snappy.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/force-sync-many.yaml workloads/snaps-few-objects.yaml} ----------------------------------------------------------------- time: 00:51:03 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062865/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062865/ SSH connection to smithi115 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max- ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5062866] rados/singleton-nomsgr/{all/recovery-unfound-found.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062866/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062866/ [5062868] rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/few.yaml objectstore/bluestore-comp-snappy.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-radosbench.yaml} ----------------------------------------------------------------- time: 00:36:42 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062868/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062868/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=a8de571e8d0f435194d3123cd47f8ef1 SSH connection to smithi018 was lost: "/bin/sh -c 'adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 -p unique_pool_1 bench 150 write'" [5062869] rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/few.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-small-objects-fast-read-overwrites.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062869/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062869/ [5062871] rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-comp-snappy.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062871/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062871/ [5062872] rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} ----------------------------------------------------------------- time: 00:36:26 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062872/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062872/ Unable to open channel. [5062873] rados/singleton/{all/osd-recovery-incomplete.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062873/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062873/ [5062877] rados/dashboard/{clusters/{2-node-mgr.yaml} debug/mgr.yaml objectstore/bluestore-comp-snappy.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/dashboard.yaml} ----------------------------------------------------------------- time: 00:51:47 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062877/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062877/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=4911b59215904bc7912ae94d6686d15f Test failure: setUpClass (tasks.mgr.dashboard.test_host.HostControllerTest) [5062878] rados/cephadm/with-work/{distro/ubuntu_latest.yaml fixed-2.yaml mode/packaged.yaml msgr/async-v2only.yaml start.yaml tasks/rados_python.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062878/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062878/ [5062879] rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062879/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062879/ [5062880] rados/monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/force-sync-many.yaml workloads/rados_mon_workunits.yaml} ----------------------------------------------------------------- time: 00:51:42 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062880/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062880/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=a58626a3bc784288ab0c97251c03e466 SSH connection to smithi084 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [5062881] rados/perf/{ceph.yaml objectstore/bluestore-bitmap.yaml openstack.yaml settings/optimized.yaml ubuntu_latest.yaml workloads/radosbench_omap_write.yaml} ----------------------------------------------------------------- time: 00:35:53 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062881/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062881/ unexpected end of data [5062882] rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/fastclose.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-small-objects-fast-read-overwrites.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062882/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062882/ Pass ================================================================= [5062841] rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/osd-delay.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-small-objects-many-deletes.yaml} time: 00:19:59 info: http://pulpito.ceph.com/kchai-2020-05-17_05:02:01-rados-wip-kefu-testing-2020-05-15-0016-distro-basic-smithi/5062841/ 2020-05-17T21:55:47.898 INFO:teuthology.results:starting coverage generation 2020-05-17T21:55:47.904 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main int(args['--timeout']), args['--dry-run']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results generate_coverage(archive_dir, name) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage archive_dir, File "/usr/lib/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage': '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage'