2020-08-29T23:32:46.402 INFO:root:teuthology version: 1.0.0-018d708 2020-08-29T23:32:46.404 INFO:teuthology.results:subset: '860/9999' 2020-08-29T23:32:46.404 INFO:teuthology.results:seed: '2477' 2020-08-29T23:40:38.535 INFO:root:teuthology version: 1.0.0-018d708 2020-08-29T23:40:38.566 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-30T02:50:43.295 INFO:teuthology.results:Tests finished! gathering results... 2020-08-30T02:50:44.863 INFO:scrape:Found 13 jobs 2020-08-30T02:50:46.965 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4531-g08fab7a8a91 (08fab7a8a9103f87935c685c0a66d28e361bc9f5) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f06fc2ae257] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x5614c8af9a60] 2020-08-30T02:50:46.968 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-29T23:51:54.186 INFO:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-29T23:51:54.451 INFO:teuthology.orchestra.run.smithi084.stdout: 2020-08-30T02:50:46.969 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T23:52:56.377 INFO:tasks.ceph:Checking cluster log for badness... , 2020-08-29T23:52:56.378 INFO:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 , 2020-08-29T23:52:56.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi084.front.sepia.ceph.com 2020-08-30T02:50:47.121 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4531-g08fab7a8a91 (08fab7a8a9103f87935c685c0a66d28e361bc9f5) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f06fc2ae257] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x5614c8af9a60] 2020-08-30T02:50:47.125 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-29T23:51:54.186 INFO:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-29T23:51:54.451 INFO:teuthology.orchestra.run.smithi084.stdout: 2020-08-30T02:50:47.126 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T23:52:56.377 INFO:tasks.ceph:Checking cluster log for badness... , 2020-08-29T23:52:56.378 INFO:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 , 2020-08-29T23:52:56.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi084.front.sepia.ceph.com 2020-08-30T02:50:47.263 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4531-g08fab7a8a91 (08fab7a8a9103f87935c685c0a66d28e361bc9f5) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f06fc2ae257] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x5614c8af9a60] 2020-08-30T02:50:47.266 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-29T23:51:54.186 INFO:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-29T23:51:54.451 INFO:teuthology.orchestra.run.smithi084.stdout: 2020-08-30T02:50:47.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T23:52:56.377 INFO:tasks.ceph:Checking cluster log for badness... , 2020-08-29T23:52:56.378 INFO:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 , 2020-08-29T23:52:56.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi084.front.sepia.ceph.com 2020-08-30T02:50:47.401 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4531-g08fab7a8a91 (08fab7a8a9103f87935c685c0a66d28e361bc9f5) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f06fc2ae257] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x5614c8af9a60] 2020-08-30T02:50:47.403 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-29T23:51:54.186 INFO:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-29T23:51:54.451 INFO:teuthology.orchestra.run.smithi084.stdout: 2020-08-30T02:50:47.404 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T23:52:56.377 INFO:tasks.ceph:Checking cluster log for badness... , 2020-08-29T23:52:56.378 INFO:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 , 2020-08-29T23:52:56.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi084.front.sepia.ceph.com 2020-08-30T02:50:47.421 INFO:scrape:Found 2 distinct failure reasons 2020-08-30T02:50:47.423 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-08-30T02:50:47.423 INFO:scrape:1 jobs: ['5388623'] 2020-08-30T02:50:47.424 INFO:scrape:suites: ['1-install/luminous-v1only', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{centos_7.6}', 'msgr-failures/few', 'rados', 'rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/snaps-few-objects}'] 2020-08-30T02:50:47.424 INFO:scrape: 2020-08-30T02:50:47.425 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-08-30T02:50:47.425 INFO:scrape:1 jobs: ['5388626'] 2020-08-30T02:50:47.425 INFO:scrape:suites: ['msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}'] 2020-08-30T02:50:47.426 INFO:scrape: 2020-08-30T02:50:47.427 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [5388623] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:23:49 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388623/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388623/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=3f4d1f3f857c4996bc03607c4223a68a reached maximum tries (180) after waiting for 180 seconds [5388626] rados/singleton/{all/thrash_cache_writeback_proxy_none msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:11:18 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388626/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388626/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' Pass ================================================================= [5388620] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:24:23 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388620/ [5388621] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:13:48 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388621/ [5388622] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:54 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388622/ [5388624] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:21:32 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388624/ [5388625] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:51 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388625/ [5388627] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:42:38 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388627/ [5388628] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:11 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388628/ [5388629] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:15 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388629/ [5388630] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} time: 03:06:14 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388630/ [5388631] rados/singleton-nomsgr/{all/admin_socket_output rados supported-random-distro$/{rhel_8}} time: 00:17:53 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388631/ [5388632] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:29:16 info: http://pulpito.ceph.com/trociny-2020-08-29_19:23:54-rados-wip-mgolub-testing-4-distro-basic-smithi/5388632/