2021-01-11T16:27:20.106 INFO:root:teuthology version: 1.0.0-77817aa4 2021-01-11T16:27:20.107 INFO:teuthology.results:subset: '300/9999' 2021-01-11T16:27:20.107 INFO:teuthology.results:seed: '626' 2021-01-11T16:35:09.966 INFO:root:teuthology version: 1.0.0-77817aa4 2021-01-11T16:35:09.993 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-12T04:35:22.581 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-01-12T04:35:23.244 INFO:scrape:Found 10 jobs 2021-01-12T04:35:25.354 INFO:scrape:Found 3 distinct failure reasons 2021-01-12T04:35:25.355 INFO:scrape:Failure: "2021-01-11T16:52:07.944354+0000 mon.a (mon.0) 156 : cluster [WRN] Health check update: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log 2021-01-12T04:35:25.355 INFO:scrape:3 jobs: ['5777746', '5777747', '5777744'] 2021-01-12T04:35:25.355 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'thrashosds-health'] 2021-01-12T04:35:25.356 INFO:scrape:suites union: ['clusters/{fixed-2', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashers/fastread', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-many-deletes}', 'workloads/ec-small-objects}'] 2021-01-12T04:35:25.356 INFO:scrape: 2021-01-12T04:35:25.357 INFO:scrape:Dead: b'2021-01-12T04:34:43.635 DEBUG:teuthology.exit:Finished running handlers' 2021-01-12T04:35:25.357 INFO:scrape:6 jobs: ['5777749', '5777741', '5777740', '5777745', '5777742', '5777748'] 2021-01-12T04:35:25.358 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'thrashosds-health'] 2021-01-12T04:35:25.358 INFO:scrape:suites union: ['clusters/{fixed-2', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-small-objects-fast-read}'] 2021-01-12T04:35:25.359 INFO:scrape: 2021-01-12T04:35:25.359 INFO:scrape:Failure: Command failed on smithi188 with status 6: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0" 2021-01-12T04:35:25.359 INFO:scrape:1 jobs: ['5777743'] 2021-01-12T04:35:25.360 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/fast', 'mon_election/connectivity', 'msgr-failures/fastclose', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-radosbench}'] 2021-01-12T04:35:25.360 INFO:scrape: 2021-01-12T04:35:25.361 INFO:teuthology.results:Sending results to dupadhya@redhat.com: Test Run: ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/ failed: 4 dead: 5 running: 1 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [5777743] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-radosbench} ----------------------------------------------------------------- time: 00:10:30 info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777743/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777743/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e0f0b326fa64d608f6640503812b953 Command failed on smithi188 with status 6: "sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0" [5777744] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-balanced} ----------------------------------------------------------------- time: 00:27:49 info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777744/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777744/ "2021-01-11T16:48:11.533911+0000 mon.b (mon.1) 59 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log [5777746] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:31:09 info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777746/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777746/ "2021-01-11T16:52:07.944354+0000 mon.a (mon.0) 156 : cluster [WRN] Health check update: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [5777747] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:30:11 info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777747/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777747/ "2021-01-11T16:48:26.320582+0000 mon.a (mon.0) 31 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log Dead ================================================================= [5777740] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777740/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777740/ [5777741] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777741/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777741/ [5777742] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777742/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777742/ [5777745] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777745/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777745/ [5777748] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777748/ log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777748/ Running ================================================================= [5777749] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} info: http://pulpito.ceph.com/ideepika-2021-01-11_13:42:06-rados:thrash-erasure-code-master-distro-basic-smithi/5777749/