2020-08-30T16:47:21.275 INFO:root:teuthology version: 1.0.0-018d708 2020-08-30T16:47:21.278 INFO:teuthology.results:seed: '9978' 2020-08-30T17:33:27.766 INFO:root:teuthology version: 1.0.0-018d708 2020-08-30T17:33:27.916 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-31T05:14:48.926 INFO:teuthology.results:Tests finished! gathering results... 2020-08-31T05:15:03.031 INFO:scrape:Found 96 jobs 2020-08-31T05:15:06.221 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-osd.5.log.gz 2020-08-31T05:15:06.222 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-osd.4.log.gz 2020-08-31T05:15:06.223 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-mgr.x.log.gz 2020-08-31T05:15:06.223 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-osd.2.log.gz 2020-08-31T05:15:06.224 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-osd.6.log.gz 2020-08-31T05:15:06.225 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-osd.7.log.gz 2020-08-31T05:15:06.225 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-mgr.y.log.gz 2020-08-31T05:15:06.226 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-osd.1.log.gz 2020-08-31T05:15:06.226 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-osd.3.log.gz 2020-08-31T05:15:06.227 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-osd.0.log.gz 2020-08-31T05:15:06.228 WARNING:scrape:Not-understood crash indication 2020-08-31T05:15:07.284 INFO:scrape:Found 6 distinct failure reasons 2020-08-31T05:15:07.286 INFO:scrape:Failure: "2020-08-30T17:12:16.987267+0000 mon.b (mon.0) 38 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log 2020-08-31T05:15:07.286 INFO:scrape:2 jobs: ['5384846', '5384916'] 2020-08-31T05:15:07.287 INFO:scrape:suites intersection: ['conf'] 2020-08-31T05:15:07.287 INFO:scrape:suites union: ['clusters/3-node', 'clusters/fixed-3', 'conf', 'features/object-map', 'krbd/fsx/{ceph/ceph', 'krbd/rbd/{bluestore-bitmap', 'msgr-failures/many', 'objectstore/filestore-xfs', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-3-client}', 'tasks/rbd_workunit_trivial_sync}'] 2020-08-31T05:15:07.288 INFO:scrape: 2020-08-31T05:15:07.288 INFO:scrape:Failure: reached maximum tries (60) after waiting for 900 seconds 2020-08-31T05:15:07.288 INFO:scrape:2 jobs: ['5384842', '5384909'] 2020-08-31T05:15:07.289 INFO:scrape: 2020-08-31T05:15:07.289 INFO:scrape:Crash: Command failed (workunit test rbd/huge-tickets.sh) on smithi188 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed90c0768b0c8cf09511e8971bba453e09e42697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/huge-tickets.sh' 2020-08-31T05:15:07.290 INFO:scrape:ceph version 16.0.0-4846-ged90c0768b (ed90c0768b0c8cf09511e8971bba453e09e42697) pacific (dev) 1: (()+0x128a0) [0x7f7c2b52b8a0] 2: (gsignal()+0xc7) [0x7f7c2a623f47] 3: (abort()+0x141) [0x7f7c2a6258b1] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f7c2c6b1e1a] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f7c2c6b1fa4] 6: (ProtocolV2::write(std::__cxx11::basic_string, std::allocator > const&, CtFun&, ceph::buffer::v15_2_0::list&)+0x1ac) [0x7f7c2c97a37c] 7: (Ct* ProtocolV2::write(std::__cxx11::basic_string, std::allocator > const&, CtFun&, ceph::msgr::v2::AuthDoneFrame&)+0x18b) [0x7f7c2c9967db] 8: (ProtocolV2::_handle_auth_request(ceph::buffer::v15_2_0::list&, bool)+0x34a) [0x7f7c2c986a5a] 9: (ProtocolV2::handle_auth_request_more(ceph::buffer::v15_2_0::list&)+0x23e) [0x7f7c2c9877fe] 10: (ProtocolV2::handle_frame_payload()+0xfb) [0x7f7c2c98fc8b] 11: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x7f7c2c98fff0] 12: (ProtocolV2::_handle_read_frame_epilogue_main()+0x69) [0x7f7c2c990159] 13: (ProtocolV2::_handle_read_frame_segment()+0x121) [0x7f7c2c990511] 14: (ProtocolV2::handle_read_frame_segment(std::unique_ptr&&, int)+0xd7) [0x7f7c2c9916f7] 15: (ProtocolV2::run_continuation(Ct&)+0x34) [0x7f7c2c979c64] 16: (AsyncConnection::process()+0x5fc) [0x7f7c2c944dfc] 17: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0x7dd) [0x7f7c2c99ab5d] 18: (()+0x554920) [0x7f7c2c9a2920] 19: (std::function::operator()() const+0x32) [0x55cbea1b7e1c] 20: (void std::__invoke_impl>(std::__invoke_other, std::function&&)+0x20) [0x55cbea551511] 21: (std::__invoke_result>::type std::__invoke>(std::function&&)+0x35) [0x55cbea55068d] 22: (decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker > >::_M_invoke<0ul>(std::_Index_tuple<0ul>)+0x28) [0x55cbea55254c] 23: (std::thread::_Invoker > >::operator()()+0x2c) [0x55cbea552508] 24: (std::thread::_State_impl > > >::_M_run()+0x1c) [0x55cbea5524d8] 25: (()+0xbd6df) [0x7f7c2b0496df] 26: (()+0x76db) [0x7f7c2b5206db] 27: (clone()+0x3f) [0x7f7c2a706a3f] 2020-08-31T05:15:07.290 INFO:scrape:2 jobs: ['5384904', '5384855'] 2020-08-31T05:15:07.291 INFO:scrape:suites intersection: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'tasks/rbd_huge_tickets}'] 2020-08-31T05:15:07.291 INFO:scrape:suites union: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'msgr-failures/few', 'msgr-failures/many', 'tasks/rbd_huge_tickets}'] 2020-08-31T05:15:07.291 INFO:scrape: 2020-08-31T05:15:07.292 INFO:scrape:Failure: Command failed on smithi098 with status 3: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludeh8kcfytg -r -- '-g auto -g blockdev -x clone'" 2020-08-31T05:15:07.292 INFO:scrape:2 jobs: ['5384876', '5384827'] 2020-08-31T05:15:07.293 INFO:scrape:suites intersection: ['conf', 'krbd/singleton/{bluestore-bitmap', 'tasks/rbd_xfstests}'] 2020-08-31T05:15:07.293 INFO:scrape:suites union: ['conf', 'krbd/singleton/{bluestore-bitmap', 'msgr-failures/few', 'msgr-failures/many', 'tasks/rbd_xfstests}'] 2020-08-31T05:15:07.293 INFO:scrape: 2020-08-31T05:15:07.294 INFO:scrape:Dead: b'2020-08-31T05:11:47.593 DEBUG:teuthology.exit:Finished running handlers' 2020-08-31T05:15:07.294 INFO:scrape:1 jobs: ['5384879'] 2020-08-31T05:15:07.295 INFO:scrape:suites: ['clusters/3-node', 'conf', 'features/no-object-map', 'krbd/fsx/{ceph/ceph', 'objectstore/filestore-xfs', 'randomized-striping-on}', 'striping/fancy/{msgr-failures/few', 'tasks/fsx-3-client}'] 2020-08-31T05:15:07.295 INFO:scrape: 2020-08-31T05:15:07.296 INFO:scrape:Failure: "2020-08-30T17:48:46.655835+0000 osd.3 (osd.3) 3 : cluster [WRN] slow request osd_op(client.4444.0:8611 2.2 2:52b97c0e:::rbd_data.11532a8deda0.00000000000002cf:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~28672 in=28672b] snapc 0=[] ondisk+write+known_if_redirected e23) initiated 2020-08-30T17:48:16.346885+0000 currently waiting for sub ops" in cluster log 2020-08-31T05:15:07.296 INFO:scrape:1 jobs: ['5384906'] 2020-08-31T05:15:07.296 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'krbd/rbd/{bluestore-bitmap', 'msgr-failures/many', 'tasks/rbd_workunit_suites_fsx}'] 2020-08-31T05:15:07.297 INFO:scrape: 2020-08-31T05:15:07.302 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/ failed: 7 dead: 3 running: 0 waiting: 0 queued: 0 passed: 86 Fail ================================================================= [5384827] krbd/singleton/{bluestore-bitmap conf msgr-failures/few tasks/rbd_xfstests} ----------------------------------------------------------------- time: 08:44:01 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384827/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384827/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=b359da396ed44862bcbb80d2b5b1dc4b Command failed on smithi076 with status 3: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude8a70byb0 -r -- '-g auto -g blockdev -x clone'" [5384846] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} ----------------------------------------------------------------- time: 00:33:38 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384846/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384846/ "2020-08-30T17:12:16.987267+0000 mon.b (mon.0) 38 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log [5384855] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/rbd_huge_tickets} ----------------------------------------------------------------- time: 00:14:50 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384855/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384855/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=b3e9335bc24c41b6a388dafba7b5728e Command failed (workunit test rbd/huge-tickets.sh) on smithi198 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed90c0768b0c8cf09511e8971bba453e09e42697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/huge-tickets.sh' [5384876] krbd/singleton/{bluestore-bitmap conf msgr-failures/many tasks/rbd_xfstests} ----------------------------------------------------------------- time: 10:05:22 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384876/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384876/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=6c9a858d27654c0e960a80dd1c08912b Command failed on smithi098 with status 3: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludeh8kcfytg -r -- '-g auto -g blockdev -x clone'" [5384904] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/rbd_huge_tickets} ----------------------------------------------------------------- time: 00:14:50 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384904/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384904/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=cadd9078e1fe4929a367a9027cd6ab95 Command failed (workunit test rbd/huge-tickets.sh) on smithi188 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed90c0768b0c8cf09511e8971bba453e09e42697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/huge-tickets.sh' [5384906] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:18:23 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384906/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384906/ "2020-08-30T17:48:46.655835+0000 osd.3 (osd.3) 3 : cluster [WRN] slow request osd_op(client.4444.0:8611 2.2 2:52b97c0e:::rbd_data.11532a8deda0.00000000000002cf:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~28672 in=28672b] snapc 0=[] ondisk+write+known_if_redirected e23) initiated 2020-08-30T17:48:16.346885+0000 currently waiting for sub ops" in cluster log [5384916] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/many tasks/rbd_workunit_trivial_sync} ----------------------------------------------------------------- time: 00:13:13 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384916/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384916/ "2020-08-30T18:23:03.689159+0000 mon.a (mon.0) 41 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log Dead ================================================================= [5384842] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384842/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384842/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=8549918a982745fb92f31dbcda48dff5 reached maximum tries (60) after waiting for 900 seconds [5384879] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/ [5384909] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/krbd_whole_object_zeroout} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384909/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384909/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=bf5972ca2f0344b19e86222494d6ffd0 reached maximum tries (60) after waiting for 900 seconds Pass ================================================================= [5384823] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/krbd_deep_flatten} time: 00:08:37 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384823/ [5384824] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:28:25 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384824/ [5384825] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/few tasks/rbd_fio} time: 00:43:41 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384825/ [5384826] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/krbd_data_pool} time: 00:21:15 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384826/ [5384828] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf thrashers/backoff thrashosds-health workloads/rbd_fio} time: 01:12:22 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384828/ [5384829] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} time: 00:06:53 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384829/ [5384830] krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_pages_required} time: 00:06:54 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384830/ [5384831] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/krbd_exclusive_option} time: 00:11:10 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384831/ [5384832] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:27:18 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384832/ [5384833] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:14:16 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384833/ [5384834] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/krbd_fallocate} time: 00:08:44 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384834/ [5384835] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:29:15 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384835/ [5384836] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/krbd_huge_osdmap} time: 00:09:42 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384836/ [5384837] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:49:01 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384837/ [5384838] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/krbd_discard} time: 00:08:11 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384838/ [5384839] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:25:59 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384839/ [5384840] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:12:56 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384840/ [5384841] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:12:16 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384841/ [5384843] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:20:53 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384843/ [5384844] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/krbd_namespaces} time: 00:11:41 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384844/ [5384845] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/krbd_udev_enumerate} time: 00:27:17 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384845/ [5384847] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:08:27 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384847/ [5384848] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/krbd_udev_netlink_enobufs} time: 00:13:54 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384848/ [5384849] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/krbd_huge_image} time: 00:06:28 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384849/ [5384850] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:22:53 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384850/ [5384851] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf thrashers/pggrow thrashosds-health workloads/rbd_fio} time: 01:21:27 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384851/ [5384852] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/rbd_concurrent} time: 00:14:38 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384852/ [5384853] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:09:08 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384853/ [5384854] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:30:40 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384854/ [5384856] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:28:27 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384856/ [5384857] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:16:11 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384857/ [5384858] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/rbd_image_read} time: 00:08:32 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384858/ [5384859] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} time: 00:07:06 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384859/ [5384860] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} time: 00:20:23 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384860/ [5384861] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/rbd_kernel} time: 00:07:51 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384861/ [5384862] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/krbd_modprobe} time: 00:07:22 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384862/ [5384863] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:23:28 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384863/ [5384864] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:58:25 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384864/ [5384865] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/many tasks/rbd_workunit_suites_iozone} time: 00:11:53 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384865/ [5384866] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:08:02 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384866/ [5384867] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:31:59 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384867/ [5384868] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/rbd_map_unmap} time: 00:12:30 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384868/ [5384869] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/few tasks/rbd_workunit_trivial_sync} time: 00:08:38 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384869/ [5384870] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:27:54 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384870/ [5384871] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/rbd_simple_big} time: 00:34:05 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384871/ [5384872] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/krbd_msgr_segments} time: 00:07:02 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384872/ [5384873] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:29:23 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384873/ [5384874] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/many tasks/rbd_fio} time: 00:39:47 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384874/ [5384875] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/krbd_data_pool} time: 00:26:10 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384875/ [5384877] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:35:00 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384877/ [5384878] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/krbd_exclusive_option} time: 00:09:43 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384878/ [5384880] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/few tasks/rbd_workunit_kernel_untar_build} time: 00:14:47 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384880/ [5384881] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/krbd_fallocate} time: 00:09:34 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384881/ [5384882] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:30:58 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384882/ [5384883] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/krbd_huge_osdmap} time: 00:09:47 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384883/ [5384884] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/many tasks/rbd_workunit_suites_dbench} time: 00:43:51 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384884/ [5384885] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/krbd_parent_overlap} time: 00:06:30 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384885/ [5384886] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:19:42 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384886/ [5384887] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} time: 01:30:54 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384887/ [5384888] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/krbd_latest_osdmap_on_map} time: 00:12:43 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384888/ [5384889] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:32:09 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384889/ [5384890] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:14:22 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384890/ [5384891] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/krbd_namespaces} time: 00:11:38 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384891/ [5384892] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} time: 00:07:03 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384892/ [5384893] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} time: 00:20:32 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384893/ [5384894] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/krbd_udev_enumerate} time: 00:37:03 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384894/ [5384895] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:26:55 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384895/ [5384896] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/many tasks/rbd_workunit_suites_fsstress} time: 00:08:42 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384896/ [5384897] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:14:05 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384897/ [5384898] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/krbd_read_only} time: 00:07:21 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384898/ [5384899] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:27:05 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384899/ [5384900] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:37:43 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384900/ [5384901] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/rbd_concurrent} time: 00:14:43 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384901/ [5384902] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:08:21 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384902/ [5384903] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:33:13 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384903/ [5384905] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:32:08 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384905/ [5384907] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/rbd_image_read} time: 00:08:31 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384907/ [5384908] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/rbd_kernel} time: 00:07:47 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384908/ [5384910] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:23:18 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384910/ [5384911] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf thrashers/upmap thrashosds-health workloads/rbd_fio} time: 01:22:33 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384911/ [5384912] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:11:20 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384912/ [5384913] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:07:40 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384913/ [5384914] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:30:38 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384914/ [5384915] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/few tasks/rbd_map_unmap} time: 00:12:50 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384915/ [5384917] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:27:49 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384917/ [5384918] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph msgr-failures/many tasks/rbd_simple_big} time: 00:35:27 info: http://pulpito.ceph.com/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384918/