2020-09-02T07:16:43.414 INFO:root:teuthology version: 1.0.0-f1fe128 2020-09-02T07:16:43.416 INFO:teuthology.results:seed: '5265' 2020-09-02T08:57:34.030 INFO:root:teuthology version: 1.0.0-f1fe128 2020-09-02T08:57:34.053 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-09-02T11:11:36.803 INFO:teuthology.results:Tests finished! gathering results... 2020-09-02T11:11:40.384 INFO:scrape:Found 40 jobs 2020-09-02T11:11:40.754 INFO:scrape:Found 2 distinct failure reasons 2020-09-02T11:11:40.756 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi057 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d74f91a305ee780f50d1ebb9cc2494847d1708f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' 2020-09-02T11:11:40.756 INFO:scrape:1 jobs: ['5399097'] 2020-09-02T11:11:40.757 INFO:scrape:suites: ['clusters/{fixed-3', 'config/none', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'pool/none', 'rbd/librbd/{cache/writearound', 'supported-random-distro$/{centos_8}', 'workloads/c_api_tests_with_defaults}'] 2020-09-02T11:11:40.757 INFO:scrape: 2020-09-02T11:11:40.758 INFO:scrape:Failure: "2020-09-02T08:11:53.914669+0000 mon.b (mon.0) 541 : cluster [WRN] Health check failed: Degraded data redundancy: 1/14 objects degraded (7.143%), 1 pg degraded (PG_DEGRADED)" in cluster log 2020-09-02T11:11:40.758 INFO:scrape:1 jobs: ['5399103'] 2020-09-02T11:11:40.759 INFO:scrape:suites: ['clusters/{fixed-3', 'config/permit-partial-discard', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'pool/none', 'rbd/librbd/{cache/writearound', 'supported-random-distro$/{rhel_8}', 'workloads/c_api_tests_with_defaults}'] 2020-09-02T11:11:40.759 INFO:scrape: 2020-09-02T11:11:40.762 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 38 Fail ================================================================= [5399097] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-stupid pool/none supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults} ----------------------------------------------------------------- time: 00:13:20 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399097/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399097/ sentry: https://sentry.ceph.com/sepia/teuthology/?query=2e9a776d1be9485091468d478e4e2cd4 Command failed (workunit test rbd/test_librbd.sh) on smithi057 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d74f91a305ee780f50d1ebb9cc2494847d1708f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=61 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' [5399103] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} ----------------------------------------------------------------- time: 01:14:48 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399103/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399103/ "2020-09-02T08:11:53.914669+0000 mon.b (mon.0) 541 : cluster [WRN] Health check failed: Degraded data redundancy: 1/14 objects degraded (7.143%), 1 pg degraded (PG_DEGRADED)" in cluster log Pass ================================================================= [5399085] rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults} time: 01:09:39 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399085/ [5399086] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 01:06:21 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399086/ [5399087] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-bitmap validator/memcheck workloads/c_api_tests} time: 00:40:53 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399087/ [5399088] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 01:14:25 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399088/ [5399089] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-comp-snappy pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} time: 01:11:37 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399089/ [5399090] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/filestore-xfs pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 01:05:47 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399090/ [5399091] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-bitmap pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 01:14:38 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399091/ [5399092] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-lz4 pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} time: 01:07:20 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399092/ [5399093] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-stupid pool/ec-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests} time: 01:06:21 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399093/ [5399094] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/filestore-xfs pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 01:14:16 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399094/ [5399095] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} time: 01:06:43 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399095/ [5399096] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/ec-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests} time: 01:06:26 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399096/ [5399098] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/filestore-xfs pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} time: 01:11:39 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399098/ [5399099] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-hybrid pool/ec-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests} time: 01:06:31 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399099/ [5399100] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 01:14:55 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399100/ [5399101] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-stupid pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests_with_journaling} time: 01:07:39 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399101/ [5399102] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests} time: 01:10:59 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399102/ [5399104] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} time: 01:06:58 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399104/ [5399105] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zlib pool/ec-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests} time: 01:06:03 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399105/ [5399106] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 01:14:08 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399106/ [5399107] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-hybrid pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} time: 01:12:05 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399107/ [5399108] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-comp-snappy pool/ec-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests} time: 01:06:17 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399108/ [5399109] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zlib pool/none supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults} time: 01:10:22 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399109/ [5399110] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zstd pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} time: 01:06:57 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399110/ [5399111] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-lz4 pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 01:06:17 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399111/ [5399112] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-snappy pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 01:14:18 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399112/ [5399113] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} time: 01:11:24 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399113/ [5399114] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-bitmap pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests} time: 01:10:05 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399114/ [5399115] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 01:11:01 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399115/ [5399116] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-comp-snappy pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} time: 01:15:25 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399116/ [5399117] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_8} workloads/c_api_tests} time: 01:09:30 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399117/ [5399118] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/c_api_tests_with_defaults} time: 01:10:32 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399118/ [5399119] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling} time: 01:10:12 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399119/ [5399120] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/c_api_tests} time: 01:14:33 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399120/ [5399121] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/filestore-xfs pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults} time: 01:06:04 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399121/ [5399122] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-bitmap pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests_with_journaling} time: 01:14:19 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399122/ [5399123] rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/c_api_tests} time: 01:14:58 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399123/ [5399124] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-lz4 validator/memcheck workloads/c_api_tests_with_defaults} time: 00:44:09 info: http://pulpito.ceph.com/trociny-2020-09-02_06:44:21-rbd-wip-mgolub-testing-distro-basic-smithi/5399124/