2020-08-29T12:35:37.844 INFO:root:teuthology version: 1.0.0-018d708 2020-08-29T12:35:37.846 INFO:teuthology.results:subset: '860/9999' 2020-08-29T12:35:37.846 INFO:teuthology.results:seed: '2477' 2020-08-29T12:43:37.396 INFO:root:teuthology version: 1.0.0-018d708 2020-08-29T12:43:37.401 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-29T15:50:42.030 INFO:teuthology.results:Tests finished! gathering results... 2020-08-29T15:50:43.584 INFO:scrape:Found 13 jobs 2020-08-29T15:50:44.024 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4532-g1f8b988e9f4 (1f8b988e9f488b48ad9aec9ecad7e29cc8151e07) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f7305706ba9] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f7305706d84] 2020-08-29T15:50:44.027 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-29T13:02:51.589 INFO:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-29T13:02:51.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2020-08-29T15:50:44.028 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T13:04:33.746 INFO:teuthology.misc:Shutting down osd daemons... , 2020-08-29T13:04:33.747 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-08-29T13:04:33.747 INFO:teuthology.orchestra.run:waiting for 300 2020-08-29T15:50:44.029 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T13:09:43.717 INFO:teuthology.task.clock:Checking final clock skew... , 2020-08-29T13:09:43.718 INFO:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , 2020-08-29T13:09:43.740 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2020-08-29T15:50:44.261 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4532-g1f8b988e9f4 (1f8b988e9f488b48ad9aec9ecad7e29cc8151e07) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f7305706ba9] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f7305706d84] 2020-08-29T15:50:44.265 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-29T13:02:51.589 INFO:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-29T13:02:51.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2020-08-29T15:50:44.266 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T13:04:33.746 INFO:teuthology.misc:Shutting down osd daemons... , 2020-08-29T13:04:33.747 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-08-29T13:04:33.747 INFO:teuthology.orchestra.run:waiting for 300 2020-08-29T15:50:44.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T13:09:43.717 INFO:teuthology.task.clock:Checking final clock skew... , 2020-08-29T13:09:43.718 INFO:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , 2020-08-29T13:09:43.740 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2020-08-29T15:50:44.478 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4532-g1f8b988e9f4 (1f8b988e9f488b48ad9aec9ecad7e29cc8151e07) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f7305706ba9] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f7305706d84] 2020-08-29T15:50:44.481 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-29T13:02:51.589 INFO:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-29T13:02:51.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2020-08-29T15:50:44.482 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T13:04:33.746 INFO:teuthology.misc:Shutting down osd daemons... , 2020-08-29T13:04:33.747 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-08-29T13:04:33.747 INFO:teuthology.orchestra.run:waiting for 300 2020-08-29T15:50:44.483 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T13:09:43.717 INFO:teuthology.task.clock:Checking final clock skew... , 2020-08-29T13:09:43.718 INFO:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , 2020-08-29T13:09:43.740 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2020-08-29T15:50:44.693 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-4532-g1f8b988e9f4 (1f8b988e9f488b48ad9aec9ecad7e29cc8151e07) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f7305706ba9] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f7305706d84] 2020-08-29T15:50:44.696 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-08-29T13:02:51.589 INFO:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-08-29T13:02:51.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2020-08-29T15:50:44.697 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T13:04:33.746 INFO:teuthology.misc:Shutting down osd daemons... , 2020-08-29T13:04:33.747 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-08-29T13:04:33.747 INFO:teuthology.orchestra.run:waiting for 300 2020-08-29T15:50:44.697 WARNING:scrape:Ignoring malparsed backtrace: 2020-08-29T13:09:43.717 INFO:teuthology.task.clock:Checking final clock skew... , 2020-08-29T13:09:43.718 INFO:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , 2020-08-29T13:09:43.740 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2020-08-29T15:50:44.724 INFO:scrape:Found 2 distinct failure reasons 2020-08-29T15:50:44.727 INFO:scrape:Failure: "2020-08-29T13:39:05.855857+0000 osd.11 (osd.11) 62 : cluster [ERR] 3.fs0 shard 0(2) 3:fbcc5ac8:::smithi12927401-6:154 : missing" in cluster log 2020-08-29T15:50:44.728 INFO:scrape:1 jobs: ['5388398'] 2020-08-29T15:50:44.728 INFO:scrape:suites: ['cluster/{12-osds', 'msgr-failures/osd-delay', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] 2020-08-29T15:50:44.729 INFO:scrape: 2020-08-29T15:50:44.730 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-08-29T15:50:44.730 INFO:scrape:1 jobs: ['5388396'] 2020-08-29T15:50:44.731 INFO:scrape:suites: ['msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}'] 2020-08-29T15:50:44.731 INFO:scrape: 2020-08-29T15:50:44.733 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [5388396] rados/singleton/{all/thrash_cache_writeback_proxy_none msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:21:51 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388396/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388396/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5388398] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:26:28 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388398/ log: http://qa-proxy.ceph.com/teuthology/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388398/ "2020-08-29T13:39:05.855857+0000 osd.11 (osd.11) 62 : cluster [ERR] 3.fs0 shard 0(2) 3:fbcc5ac8:::smithi12927401-6:154 : missing" in cluster log Pass ================================================================= [5388390] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:24:41 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388390/ [5388391] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:03 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388391/ [5388392] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:07 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388392/ [5388393] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_7.6} msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:40:47 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388393/ [5388394] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:21:06 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388394/ [5388395] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:33:10 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388395/ [5388397] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_7.6} msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:34:23 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388397/ [5388399] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:13:44 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388399/ [5388400] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} time: 03:07:55 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388400/ [5388401] rados/singleton-nomsgr/{all/admin_socket_output rados supported-random-distro$/{rhel_8}} time: 00:16:36 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388401/ [5388402] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:01 info: http://pulpito.ceph.com/trociny-2020-08-29_12:34:56-rados-wip-mgolub-testing-3-distro-basic-smithi/5388402/