2022-02-02T14:57:47.903 INFO:root:teuthology version: 1.1.0-0f634dc1
2022-02-02T14:57:47.904 INFO:teuthology.results:seed: '1104'
2022-02-02T19:22:20.705 INFO:root:teuthology version: 1.1.0-0f634dc1
2022-02-02T19:22:20.741 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-02-02T19:28:21.215 INFO:teuthology.results:Tests finished! gathering results...
2022-02-02T19:28:26.129 INFO:scrape:Found 50 jobs
2022-02-02T19:28:31.058 INFO:scrape:Found 2 distinct failure reasons
2022-02-02T19:28:31.059 INFO:scrape:Failure: timeout expired in wait_until_healthy
2022-02-02T19:28:31.059 INFO:scrape:40 jobs
2022-02-02T19:28:31.060 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'thrashosds-health']
2022-02-02T19:28:31.060 INFO:scrape:suites union: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-radosbench}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects}']
2022-02-02T19:28:31.061 INFO:scrape:
2022-02-02T19:28:31.061 INFO:scrape:Failure: Command failed on smithi145 with status 1: 'sudo yum install -y kernel'
2022-02-02T19:28:31.062 INFO:scrape:10 jobs: ['6655236', '6655235', '6655231', '6655237', '6655239', '6655233', '6655232', '6655238', '6655234', '6655230']
2022-02-02T19:28:31.062 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}']
2022-02-02T19:28:31.063 INFO:scrape:suites union: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}']
2022-02-02T19:28:31.063 INFO:scrape:
2022-02-02T19:28:31.065 INFO:teuthology.results:Sending results to amathuri@redhat.com: Test Run: amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/
failed:  50
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6655190]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:21:42
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655190/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655190/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0187b8dd0c4946bda011f2e637459ff7

    timeout expired in wait_until_healthy

[6655191]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:20:02
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655191/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655191/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3c73a5a6cae54f2db1e9c971f8f28fab

    timeout expired in wait_until_healthy

[6655192]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:31:55
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655192/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655192/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d64ceed149004c089d4a5ac7ca49fadd

    timeout expired in wait_until_healthy

[6655193]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:24:33
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655193/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655193/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd94ee66783647d880416d419bea0328

    timeout expired in wait_until_healthy

[6655194]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:26:01
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655194/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655194/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22bb816d077a41c4b120f8e193001a10

    timeout expired in wait_until_healthy

[6655195]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:19:42
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655195/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655195/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9a6900c225e64f83b284e5fe492d7c43

    timeout expired in wait_until_healthy

[6655196]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:35:36
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655196/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655196/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4ba03aefe0a945688a224bcbaba69bbc

    timeout expired in wait_until_healthy

[6655197]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:22:28
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655197/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655197/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca3ab2dd851e45edbad734151f0d199d

    timeout expired in wait_until_healthy

[6655198]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:22:25
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655198/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655198/
sentry: https://sentry.ceph.com/organizations/ceph/?query=af705e6396874bedb46caedddaa7ca90

    timeout expired in wait_until_healthy

[6655199]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
-----------------------------------------------------------------
time:   00:23:49
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655199/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655199/
sentry: https://sentry.ceph.com/organizations/ceph/?query=659ca03a716a421c947bcd12a6b2cb2e

    timeout expired in wait_until_healthy

[6655200]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:18:12
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655200/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655200/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b582dfe211a14eceae17f9582e805636

    timeout expired in wait_until_healthy

[6655201]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:21:46
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655201/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655201/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f89a4a7f5aac478a87dacaeb76b2a629

    timeout expired in wait_until_healthy

[6655202]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:26:13
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655202/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655202/
sentry: https://sentry.ceph.com/organizations/ceph/?query=95e793bafef044bd8f636ca2d8da4c51

    timeout expired in wait_until_healthy

[6655203]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:22:18
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655203/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655203/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f2033699c15243e091d1b6d0b9f42b92

    timeout expired in wait_until_healthy

[6655204]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:18:04
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655204/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655204/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3dbf5b809cc14e35afd179145b6e61ee

    timeout expired in wait_until_healthy

[6655205]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:17:47
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655205/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655205/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dd9effc23b5846ab81300b5d095d3f6f

    timeout expired in wait_until_healthy

[6655206]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:16:12
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655206/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655206/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bbdba32b628240a19530918acbac66fc

    timeout expired in wait_until_healthy

[6655207]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:19:49
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655207/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655207/
sentry: https://sentry.ceph.com/organizations/ceph/?query=40106f7d5c844379af4e78f19da10be8

    timeout expired in wait_until_healthy

[6655208]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:21:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655208/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655208/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b5dad145feaf4df7986dd16387535d61

    timeout expired in wait_until_healthy

[6655209]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
-----------------------------------------------------------------
time:   00:16:08
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655209/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655209/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ffa97b728d9e4877a0f3b8c657048bce

    timeout expired in wait_until_healthy

[6655210]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:20:35
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655210/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655210/
sentry: https://sentry.ceph.com/organizations/ceph/?query=77e7d949701842edaea48b3a833e83e4

    timeout expired in wait_until_healthy

[6655211]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:20:31
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655211/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655211/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e7aabb646b76476e82d0df0b652088d3

    timeout expired in wait_until_healthy

[6655212]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:12:51
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655212/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655212/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f570bee38be34302a155503609b92ac8

    timeout expired in wait_until_healthy

[6655213]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:17:04
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655213/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655213/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f1fa9f37a84e42338b20868a660d9e8c

    timeout expired in wait_until_healthy

[6655214]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:15:25
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655214/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655214/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f1da6703c4034bf1af23440168f7e397

    timeout expired in wait_until_healthy

[6655215]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:15:48
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655215/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655215/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a48b707c15144a24a54d77641fcb18cd

    timeout expired in wait_until_healthy

[6655216]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:28:12
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655216/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655216/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ba71ff9861ab4afba9743a734366dd19

    timeout expired in wait_until_healthy

[6655217]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:15:03
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655217/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655217/
sentry: https://sentry.ceph.com/organizations/ceph/?query=467d6968b8e44652bd759fdfba5d4c53

    timeout expired in wait_until_healthy

[6655218]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:18:40
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655218/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655218/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6cca2e506d3d4d73abfc8bed5476df3e

    timeout expired in wait_until_healthy

[6655219]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:16:40
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655219/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655219/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b1ce7e2fe4b341c0aaa2f36436974958

    timeout expired in wait_until_healthy

[6655220]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:24:52
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655220/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655220/
sentry: https://sentry.ceph.com/organizations/ceph/?query=766a198d86914fbc81397405eec253c3

    timeout expired in wait_until_healthy

[6655221]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:24:49
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655221/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655221/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9619cabb3fc841e9b2267831e78c6637

    timeout expired in wait_until_healthy

[6655222]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:25:56
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655222/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655222/
sentry: https://sentry.ceph.com/organizations/ceph/?query=482ee50f63f6448ebbd4aedd6e0c7cfc

    timeout expired in wait_until_healthy

[6655223]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:25:02
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655223/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655223/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c90550813eb04f3c8c9d109ce53aa13f

    timeout expired in wait_until_healthy

[6655224]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:25:46
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655224/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655224/
sentry: https://sentry.ceph.com/organizations/ceph/?query=acd2b17b4d12422e9346ec2925858f0b

    timeout expired in wait_until_healthy

[6655225]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:25:02
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655225/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655225/
sentry: https://sentry.ceph.com/organizations/ceph/?query=41a561986fe8426da9ff607fe3e4b0cd

    timeout expired in wait_until_healthy

[6655226]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:25:57
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655226/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655226/
sentry: https://sentry.ceph.com/organizations/ceph/?query=90c777d827384e4f9b2263b59b5ea5d6

    timeout expired in wait_until_healthy

[6655227]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:25:09
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655227/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655227/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dbdd326426d04871aa9c7a8b51c4565d

    timeout expired in wait_until_healthy

[6655228]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:26:14
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655228/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655228/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e92dc603286b40e0bf25821c814e6f3f

    timeout expired in wait_until_healthy

[6655229]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
-----------------------------------------------------------------
time:   00:26:03
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655229/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655229/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2e656fa03c844740b60c495040ad9632

    timeout expired in wait_until_healthy

[6655230]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655230/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655230/
sentry: https://sentry.ceph.com/organizations/ceph/?query=311411aa362d4414860f960315a246da

    Command failed on smithi040 with status 1: 'sudo yum install -y kernel'

[6655231]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655231/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655231/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7529a814945941989e3e8972aa93d89f

    Command failed on smithi154 with status 1: 'sudo yum install -y kernel'

[6655232]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655232/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655232/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5f4b051b91934210ba0650eb910b3afc

    Command failed on smithi143 with status 1: 'sudo yum install -y kernel'

[6655233]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655233/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655233/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ba00cc59eeb549ac8d6a3eddff835062

    Command failed on smithi198 with status 1: 'sudo yum install -y kernel'

[6655234]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655234/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655234/
sentry: https://sentry.ceph.com/organizations/ceph/?query=16151a5fbd95493bbd1c5201aabf18d9

    Command failed on smithi197 with status 1: 'sudo yum install -y kernel'

[6655235]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655235/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655235/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b122c4da6d9645708e77d336762bfbfa

    Command failed on smithi087 with status 1: 'sudo yum install -y kernel'

[6655236]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655236/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655236/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3501ca7064894bff999a635e446d6b30

    Command failed on smithi145 with status 1: 'sudo yum install -y kernel'

[6655237]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655237/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655237/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cb904e297b1f455fac4b3240f942adbb

    Command failed on smithi107 with status 1: 'sudo yum install -y kernel'

[6655238]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655238/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655238/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f6a9b3b89dec4e44ae8c65b3e15ec7ef

    Command failed on smithi100 with status 1: 'sudo yum install -y kernel'

[6655239]  rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655239/
log:    http://qa-proxy.ceph.com/teuthology/amathuri-2022-02-02_07:37:40-rados-master-distro-basic-smithi/6655239/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5096efa6ab07450081fdd4f6dde44360

    Command failed on smithi043 with status 1: 'sudo yum install -y kernel'