Found 36 jobs Ignoring malparsed backtrace: ceph version 17.0.0-598-g22e68cd6 (22e68cd6c03507c7b7ee06891cbd2387672737b3) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x63fdc92] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x27beac) [0x63fdeac] Ignoring malparsed backtrace: ==56500== 32 bytes in 1 blocks are possibly lost in loss record 801 of 2,616 , ==56500== at 0x4C3321A: calloc (vg_replace_malloc.c:760) , ==56500== by 0x11F5D685: __trans_list_add (in /usr/lib64/libnl-3.so.200.26.0) Ignoring malparsed backtrace: ==56500== by 0xDB560F: testing::UnitTest::Run() (gtest.cc:5306) , ==56500== , ==56500== 224 bytes in 2 blocks are possibly lost in loss record 1,841 of 2,616 Ignoring malparsed backtrace: ==56500== by 0x4EFA185: librados::v14_2_0::RadosClient::connect() (RadosClient.cc:274) , ==56500== by 0xD94037: connect_cluster_pp(librados::v14_2_0::Rados&, std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&) (test_cxx.cc:161) , ==56500== by 0xD94C2F: connect_cluster_pp[abi:cxx11](librados::v14_2_0::Rados&) (test_cxx.cc:126) Ignoring malparsed backtrace: ==56500== by 0x4012B11: _dl_allocate_tls (in /usr/lib64/ld-2.28.so) , ==56500== by 0x51F8F26: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.28.so) , ==56500== by 0x64EEF64: Thread::try_create(unsigned long) (Thread.cc:141) Ignoring malparsed backtrace: ==56500== by 0x4EF9E47: librados::v14_2_0::RadosClient::connect() (RadosClient.cc:227) , ==56500== by 0xD94037: connect_cluster_pp(librados::v14_2_0::Rados&, std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&) (test_cxx.cc:161) , ==56500== Ignoring malparsed backtrace: /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh: line 8: 56500 Aborted valgrind ${VALGRIND} --suppressions=${TESTDIR}/valgrind.supp --error-exitcode=1 ceph_test_rbd_mirror , 2021-02-08T19:49:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 134 , 2021-02-08T19:49:27.456 INFO:tasks.workunit:Stopping ['rbd/test_rbd_mirror.sh'] on client.0... Ignoring malparsed backtrace: 2021-02-08T19:49:38.333 INFO:tasks.cephadm:Checking cluster log for badness... , 2021-02-08T19:49:38.333 DEBUG:teuthology.orchestra.run.gibba038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1738dfe-6a41-11eb-8fde-001a4aab830c/ceph.log | egrep -v '$MDS_ALL_DOWN$' | egrep -v '$MDS_UP_LESS_THAN_MAX$' | egrep -v 'overall HEALTH_' | egrep -v '$CACHE_POOL_NO_HIT_SET$' | egrep -v '$POOL_APP_NOT_ENABLED$' | head -n 1 , grep: /var/log/ceph/c1738dfe-6a41-11eb-8fde-001a4aab830c/ceph.log: No such file or directory Ignoring malparsed backtrace: ceph version 17.0.0-598-g22e68cd6 (22e68cd6c03507c7b7ee06891cbd2387672737b3) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x63fdc92] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x27beac) [0x63fdeac] Ignoring malparsed backtrace: ==56500== 32 bytes in 1 blocks are possibly lost in loss record 801 of 2,616 , ==56500== at 0x4C3321A: calloc (vg_replace_malloc.c:760) , ==56500== by 0x11F5D685: __trans_list_add (in /usr/lib64/libnl-3.so.200.26.0) Ignoring malparsed backtrace: ==56500== by 0xDB560F: testing::UnitTest::Run() (gtest.cc:5306) , ==56500== , ==56500== 224 bytes in 2 blocks are possibly lost in loss record 1,841 of 2,616 Ignoring malparsed backtrace: ==56500== by 0x4EFA185: librados::v14_2_0::RadosClient::connect() (RadosClient.cc:274) , ==56500== by 0xD94037: connect_cluster_pp(librados::v14_2_0::Rados&, std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&) (test_cxx.cc:161) , ==56500== by 0xD94C2F: connect_cluster_pp[abi:cxx11](librados::v14_2_0::Rados&) (test_cxx.cc:126) Ignoring malparsed backtrace: ==56500== by 0x4012B11: _dl_allocate_tls (in /usr/lib64/ld-2.28.so) , ==56500== by 0x51F8F26: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.28.so) , ==56500== by 0x64EEF64: Thread::try_create(unsigned long) (Thread.cc:141) Ignoring malparsed backtrace: ==56500== by 0x4EF9E47: librados::v14_2_0::RadosClient::connect() (RadosClient.cc:227) , ==56500== by 0xD94037: connect_cluster_pp(librados::v14_2_0::Rados&, std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&) (test_cxx.cc:161) , ==56500== Ignoring malparsed backtrace: /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh: line 8: 56500 Aborted valgrind ${VALGRIND} --suppressions=${TESTDIR}/valgrind.supp --error-exitcode=1 ceph_test_rbd_mirror , 2021-02-08T19:49:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 134 , 2021-02-08T19:49:27.456 INFO:tasks.workunit:Stopping ['rbd/test_rbd_mirror.sh'] on client.0... Ignoring malparsed backtrace: 2021-02-08T19:49:38.333 INFO:tasks.cephadm:Checking cluster log for badness... , 2021-02-08T19:49:38.333 DEBUG:teuthology.orchestra.run.gibba038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1738dfe-6a41-11eb-8fde-001a4aab830c/ceph.log | egrep -v '$MDS_ALL_DOWN$' | egrep -v '$MDS_UP_LESS_THAN_MAX$' | egrep -v 'overall HEALTH_' | egrep -v '$CACHE_POOL_NO_HIT_SET$' | egrep -v '$POOL_APP_NOT_ENABLED$' | head -n 1 , grep: /var/log/ceph/c1738dfe-6a41-11eb-8fde-001a4aab830c/ceph.log: No such file or directory Ignoring malparsed backtrace: ceph version 17.0.0-598-g22e68cd6 (22e68cd6c03507c7b7ee06891cbd2387672737b3) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x63fdc92] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x27beac) [0x63fdeac] Ignoring malparsed backtrace: ==56500== 32 bytes in 1 blocks are possibly lost in loss record 801 of 2,616 , ==56500== at 0x4C3321A: calloc (vg_replace_malloc.c:760) , ==56500== by 0x11F5D685: __trans_list_add (in /usr/lib64/libnl-3.so.200.26.0) Ignoring malparsed backtrace: ==56500== by 0xDB560F: testing::UnitTest::Run() (gtest.cc:5306) , ==56500== , ==56500== 224 bytes in 2 blocks are possibly lost in loss record 1,841 of 2,616 Ignoring malparsed backtrace: ==56500== by 0x4EFA185: librados::v14_2_0::RadosClient::connect() (RadosClient.cc:274) , ==56500== by 0xD94037: connect_cluster_pp(librados::v14_2_0::Rados&, std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&) (test_cxx.cc:161) , ==56500== by 0xD94C2F: connect_cluster_pp[abi:cxx11](librados::v14_2_0::Rados&) (test_cxx.cc:126) Ignoring malparsed backtrace: ==56500== by 0x4012B11: _dl_allocate_tls (in /usr/lib64/ld-2.28.so) , ==56500== by 0x51F8F26: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.28.so) , ==56500== by 0x64EEF64: Thread::try_create(unsigned long) (Thread.cc:141) Ignoring malparsed backtrace: ==56500== by 0x4EF9E47: librados::v14_2_0::RadosClient::connect() (RadosClient.cc:227) , ==56500== by 0xD94037: connect_cluster_pp(librados::v14_2_0::Rados&, std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&) (test_cxx.cc:161) , ==56500== Ignoring malparsed backtrace: /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh: line 8: 56500 Aborted valgrind ${VALGRIND} --suppressions=${TESTDIR}/valgrind.supp --error-exitcode=1 ceph_test_rbd_mirror , 2021-02-08T19:49:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 134 , 2021-02-08T19:49:27.456 INFO:tasks.workunit:Stopping ['rbd/test_rbd_mirror.sh'] on client.0... Ignoring malparsed backtrace: 2021-02-08T19:49:38.333 INFO:tasks.cephadm:Checking cluster log for badness... , 2021-02-08T19:49:38.333 DEBUG:teuthology.orchestra.run.gibba038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1738dfe-6a41-11eb-8fde-001a4aab830c/ceph.log | egrep -v '$MDS_ALL_DOWN$' | egrep -v '$MDS_UP_LESS_THAN_MAX$' | egrep -v 'overall HEALTH_' | egrep -v '$CACHE_POOL_NO_HIT_SET$' | egrep -v '$POOL_APP_NOT_ENABLED$' | head -n 1 , grep: /var/log/ceph/c1738dfe-6a41-11eb-8fde-001a4aab830c/ceph.log: No such file or directory Ignoring malparsed backtrace: ceph version 17.0.0-598-g22e68cd6 (22e68cd6c03507c7b7ee06891cbd2387672737b3) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x63fdc92] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x27beac) [0x63fdeac] Ignoring malparsed backtrace: ==56500== 32 bytes in 1 blocks are possibly lost in loss record 801 of 2,616 , ==56500== at 0x4C3321A: calloc (vg_replace_malloc.c:760) , ==56500== by 0x11F5D685: __trans_list_add (in /usr/lib64/libnl-3.so.200.26.0) Ignoring malparsed backtrace: ==56500== by 0xDB560F: testing::UnitTest::Run() (gtest.cc:5306) , ==56500== , ==56500== 224 bytes in 2 blocks are possibly lost in loss record 1,841 of 2,616 Ignoring malparsed backtrace: ==56500== by 0x4EFA185: librados::v14_2_0::RadosClient::connect() (RadosClient.cc:274) , ==56500== by 0xD94037: connect_cluster_pp(librados::v14_2_0::Rados&, std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&) (test_cxx.cc:161) , ==56500== by 0xD94C2F: connect_cluster_pp[abi:cxx11](librados::v14_2_0::Rados&) (test_cxx.cc:126) Ignoring malparsed backtrace: ==56500== by 0x4012B11: _dl_allocate_tls (in /usr/lib64/ld-2.28.so) , ==56500== by 0x51F8F26: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.28.so) , ==56500== by 0x64EEF64: Thread::try_create(unsigned long) (Thread.cc:141) Ignoring malparsed backtrace: ==56500== by 0x4EF9E47: librados::v14_2_0::RadosClient::connect() (RadosClient.cc:227) , ==56500== by 0xD94037: connect_cluster_pp(librados::v14_2_0::Rados&, std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&) (test_cxx.cc:161) , ==56500== Ignoring malparsed backtrace: /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh: line 8: 56500 Aborted valgrind ${VALGRIND} --suppressions=${TESTDIR}/valgrind.supp --error-exitcode=1 ceph_test_rbd_mirror , 2021-02-08T19:49:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 134 , 2021-02-08T19:49:27.456 INFO:tasks.workunit:Stopping ['rbd/test_rbd_mirror.sh'] on client.0... Ignoring malparsed backtrace: 2021-02-08T19:49:38.333 INFO:tasks.cephadm:Checking cluster log for badness... , 2021-02-08T19:49:38.333 DEBUG:teuthology.orchestra.run.gibba038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1738dfe-6a41-11eb-8fde-001a4aab830c/ceph.log | egrep -v '$MDS_ALL_DOWN$' | egrep -v '$MDS_UP_LESS_THAN_MAX$' | egrep -v 'overall HEALTH_' | egrep -v '$CACHE_POOL_NO_HIT_SET$' | egrep -v '$POOL_APP_NOT_ENABLED$' | head -n 1 , grep: /var/log/ceph/c1738dfe-6a41-11eb-8fde-001a4aab830c/ceph.log: No such file or directory Found 2 distinct failure reasons Failure: Command failed on gibba011 with status 1: "cd /home/ubuntu/cephtest && adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/fsx_client.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 134217728 -S 0 -N 1000 pool_client.0 image_client.0" 4 jobs: ['5868171', '5868179', '5868155', '5868163'] suites intersection: ['centos_latest', 'clusters/{fixed-1', 'openstack}', 'rbd:valgrind/{base/install', 'validator/memcheck', 'workloads/fsx}'] suites union: ['centos_latest', 'clusters/{fixed-1', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rbd:valgrind/{base/install', 'validator/memcheck', 'workloads/fsx}'] Failure: Command failed (workunit test rbd/test_rbd_mirror.sh) on gibba038 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=845271bf0003b38f37223458b7e63348cd6189e4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 VALGRIND=\'--tool=memcheck --leak-check=full\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh' 1 jobs: ['5868175'] suites: ['centos_latest', 'clusters/{fixed-1', 'objectstore/bluestore-bitmap', 'openstack}', 'rbd:valgrind/{base/install', 'validator/memcheck', 'workloads/rbd_mirror}']