2021-06-17T08:21:50.420 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-17T08:21:50.422 INFO:teuthology.results:seed: '8211' 2021-06-17T08:22:27.661 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-17T08:22:27.666 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-06-17T13:33:47.329 INFO:teuthology.results:Tests finished! gathering results... 2021-06-17T13:33:49.086 INFO:scrape:Found 20 jobs 2021-06-17T13:33:50.480 INFO:scrape:Found 3 distinct failure reasons 2021-06-17T13:33:50.480 INFO:scrape:Crash: Found coredumps on ubuntu@smithi116.front.sepia.ceph.com 2021-06-17T13:33:50.481 INFO:scrape:ceph version 14.2.21-344-g6bfb2e54a9a (6bfb2e54a9a24c38bb8cec526b0f863e562078fb) nautilus (stable) 1: (()+0xf630) [0x7fdf94722630] 2: (gsignal()+0x37) [0x7fdf92cd93d7] 3: (abort()+0x148) [0x7fdf92cdaac8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7fdf96256436] 5: (()+0x25d5af) [0x7fdf962565af] 6: (rbd::mirror::ImageReplayer::~ImageReplayer()+0x59c) [0x55781dd5a04c] 7: (rbd::mirror::ImageReplayer::~ImageReplayer()+0x9) [0x55781dd5a069] 8: (rbd::mirror::InstanceReplayer::handle_stop_image_replayers(int)+0xa2) [0x55781dd0a3d2] 9: (Context::complete(int)+0x9) [0x55781dcc7219] 10: (ThreadPool::worker(ThreadPool::WorkThread*)+0xd5c) [0x7fdf9633ca9c] 11: (ThreadPool::WorkThread::entry()+0x11) [0x7fdf9633d2b1] 12: (()+0x7ea5) [0x7fdf9471aea5] 13: (clone()+0x6d) [0x7fdf92da19fd] 2021-06-17T13:33:50.481 INFO:scrape:1 jobs: ['6176343'] 2021-06-17T13:33:50.482 INFO:scrape:suites: ['cluster/{2-node', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_7}', 'users/mirror', 'workloads/rbd-mirror-stress-workunit}'] 2021-06-17T13:33:50.482 INFO:scrape: 2021-06-17T13:33:50.482 INFO:scrape:Crash: error during thrashing 2021-06-17T13:33:50.483 INFO:scrape:ceph version 14.2.21-344-g6bfb2e54a9a (6bfb2e54a9a24c38bb8cec526b0f863e562078fb) nautilus (stable) 1: (()+0x12980) [0x7fa9bc26c980] 2: (pthread_cond_wait()+0x243) [0x7fa9bc267ad3] 3: (rbd::mirror::InstanceReplayer::shut_down()+0x19d) [0x55901eee816d] 4: (rbd::mirror::PoolReplayer::shut_down()+0x99) [0x55901eeab769] 5: (rbd::mirror::PoolReplayer::~PoolReplayer()+0xa5) [0x55901eeaea65] 6: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x3d) [0x55901eea6d7d] 7: (rbd::mirror::Mirror::~Mirror()+0xb0) [0x55901eea2d00] 8: (main()+0x3e5) [0x55901ee925b5] 9: (__libc_start_main()+0xe7) [0x7fa9bb4ddbf7] 10: (_start()+0x2a) [0x55901ee9ed5a] 2021-06-17T13:33:50.483 INFO:scrape:1 jobs: ['6176351'] 2021-06-17T13:33:50.484 INFO:scrape:suites: ['cluster/{2-node', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'users/mirror', 'workloads/rbd-mirror-fsx-workunit}'] 2021-06-17T13:33:50.484 INFO:scrape: 2021-06-17T13:33:50.484 INFO:scrape:Crash: error during thrashing 2021-06-17T13:33:50.485 INFO:scrape:ceph version 14.2.21-344-g6bfb2e54a9a (6bfb2e54a9a24c38bb8cec526b0f863e562078fb) nautilus (stable) 1: (()+0xf630) [0x7efea14c7630] 2: (pthread_cond_wait()+0xc5) [0x7efea14c3a35] 3: (rbd::mirror::InstanceReplayer::shut_down()+0x197) [0x5580ffde1b77] 4: (rbd::mirror::PoolReplayer::shut_down()+0x96) [0x5580ffda37a6] 5: (rbd::mirror::PoolReplayer::~PoolReplayer()+0xa4) [0x5580ffda7a74] 6: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x3f) [0x5580ffd9ed8f] 7: (rbd::mirror::Mirror::~Mirror()+0xaf) [0x5580ffd9a67f] 8: (main()+0x3a8) [0x5580ffd89f68] 9: (__libc_start_main()+0xf5) [0x7efe9fa6a555] 10: (()+0x214e30) [0x5580ffd96e30] 2021-06-17T13:33:50.485 INFO:scrape:1 jobs: ['6176339'] 2021-06-17T13:33:50.486 INFO:scrape:suites: ['cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_7}', 'users/mirror', 'workloads/rbd-mirror-fsx-workunit}'] 2021-06-17T13:33:50.486 INFO:scrape: 2021-06-17T13:33:50.487 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 17 Fail ================================================================= [6176339] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_7} users/mirror workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 03:27:06 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176339/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176339/ error during thrashing [6176343] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_7} users/mirror workloads/rbd-mirror-stress-workunit} ----------------------------------------------------------------- time: 01:01:45 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176343/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176343/ Found coredumps on ubuntu@smithi116.front.sepia.ceph.com [6176351] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} users/mirror workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 04:39:26 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176351/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176351/ error during thrashing Pass ================================================================= [6176336] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} users/mirror workloads/rbd-mirror-fsx-workunit} time: 04:41:43 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176336/ [6176337] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} users/mirror workloads/rbd-mirror-stress-workunit} time: 01:06:48 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176337/ [6176338] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} users/mirror workloads/rbd-mirror-workunit} time: 00:36:32 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176338/ [6176340] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} users/mirror workloads/rbd-mirror-stress-workunit} time: 01:12:54 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176340/ [6176341] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} users/mirror workloads/rbd-mirror-workunit} time: 00:40:28 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176341/ [6176342] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} users/mirror workloads/rbd-mirror-fsx-workunit} time: 05:01:28 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176342/ [6176344] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_7} users/mirror workloads/rbd-mirror-workunit} time: 00:37:32 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176344/ [6176345] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_16.04} users/mirror workloads/rbd-mirror-fsx-workunit} time: 04:18:35 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176345/ [6176346] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_16.04} users/mirror workloads/rbd-mirror-stress-workunit} time: 01:06:11 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176346/ [6176347] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} users/mirror workloads/rbd-mirror-workunit} time: 00:38:00 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176347/ [6176348] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_16.04} users/mirror workloads/rbd-mirror-fsx-workunit} time: 03:41:14 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176348/ [6176349] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_7} users/mirror workloads/rbd-mirror-stress-workunit} time: 01:05:00 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176349/ [6176350] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_16.04} users/mirror workloads/rbd-mirror-workunit} time: 00:37:00 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176350/ [6176352] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_16.04} users/mirror workloads/rbd-mirror-stress-workunit} time: 01:03:03 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176352/ [6176353] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_7} users/mirror workloads/rbd-mirror-workunit} time: 00:41:46 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176353/ [6176354] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_16.04} users/mirror workloads/rbd-mirror-fsx-workunit} time: 03:46:52 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176354/ [6176355] rbd/mirror-thrash/{base/install cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_7} users/mirror workloads/rbd-mirror-stress-workunit} time: 01:10:13 info: http://pulpito.ceph.com/trociny-2021-06-17_08:21:42-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6176355/