2020-10-23T22:01:11.617 INFO:root:teuthology version: 1.0.0-f4fe713
2020-10-23T22:01:11.618 INFO:teuthology.results:seed: '38'
2020-10-23T22:23:15.640 INFO:root:teuthology version: 1.0.0-f4fe713
2020-10-23T22:23:15.680 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-10-24T04:43:22.837 INFO:teuthology.results:Tests finished! gathering results...
2020-10-24T04:43:30.105 INFO:scrape:Found 49 jobs
2020-10-24T04:43:41.998 INFO:scrape:Found 3 distinct failure reasons
2020-10-24T04:43:41.999 INFO:scrape:Failure: timeout expired in wait_until_healthy
2020-10-24T04:43:42.000 INFO:scrape:33 jobs
2020-10-24T04:43:42.000 INFO:scrape:suites intersection: []
2020-10-24T04:43:42.001 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-client', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-upgrade', '2-workload/{blogbench', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/pacific}}', '3-thrash/default', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '3-verify}', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-compat_client', '4-ec-workload/{rados-ec-workload', '4-octopus', '4-workload/{radosbench', '5-client-sanity}}', '5-final-workload/{blogbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ec-rados-default', 'mds', 'mon', 'mon_election/classic', 'mon_election/connectivity', 'multimds/no', 'multimds/yes', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'rbd_cls', 'rbd_import_export', 'readwrite', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'tasks/{0-nautilus', 'test_rbd_api', 'test_rbd_python}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap', 'upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap', 'upgrade/cephfs/volumes/import-legacy/{bluestore-bitmap', 'upgrade/nautilus-x/parallel/{0-cluster/{openstack', 'upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade/nautilus-x/stress-split/{0-cluster/{openstack', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-10-24T04:43:42.001 INFO:scrape:
2020-10-24T04:43:42.001 INFO:scrape:Failure: "2020-10-24T01:21:07.302204+0000 mon.c (mon.0) 24 : cluster [WRN] Health check failed: There are daemons running an older version of ceph (DAEMON_OLD_VERSION)" in cluster log
2020-10-24T04:43:42.002 INFO:scrape:8 jobs: ['5552333', '5552316', '5552321', '5552291', '5552323', '5552317', '5552297', '5552315']
2020-10-24T04:43:42.002 INFO:scrape:suites intersection: ['2-partial-upgrade/firsthalf', '3-thrash/default', 'start}', 'thrashosds-health']
2020-10-24T04:43:42.003 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-workload/{radosbench', '4-workload/{rbd-cls', '5-finish-upgrade', '5-workload/{radosbench', '6-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '7-octopus', '8-final-workload/{rbd-python', '8-workload/{rbd-python', 'bluestore-bitmap', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'rbd_api}', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'ubuntu_latest}', 'upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade/nautilus-x/stress-split/{0-cluster/{openstack']
2020-10-24T04:43:42.003 INFO:scrape:
2020-10-24T04:43:42.003 INFO:scrape:Failure: list index out of range
2020-10-24T04:43:42.004 INFO:scrape:8 jobs: ['5552312', '5552329', '5552298', '5552288', '5552313', '5552322', '5552305', '5552289']
2020-10-24T04:43:42.004 INFO:scrape:suites intersection: []
2020-10-24T04:43:42.004 INFO:scrape:suites union: ['upgrade/mimic-x-singleton/mon_election/classic', 'upgrade/mimic-x-singleton/mon_election/connectivity', 'upgrade/mimic-x/parallel/mon_election/classic', 'upgrade/mimic-x/parallel/mon_election/connectivity', 'upgrade/mimic-x/stress-split-erasure-code/mon_election/classic', 'upgrade/mimic-x/stress-split-erasure-code/mon_election/connectivity', 'upgrade/mimic-x/stress-split/mon_election/classic', 'upgrade/mimic-x/stress-split/mon_election/connectivity']
2020-10-24T04:43:42.005 INFO:scrape:
2020-10-24T04:43:42.009 INFO:teuthology.results:Sending results to dzafman@redhat.com: Test Run: dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/
failed:  49
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[5552287]  upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:25:10
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552287/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552287/
sentry: https://sentry.ceph.com/sentry/sepia/?query=496870daf6e34db28efcd6f3306f2586

    timeout expired in wait_until_healthy

[5552288]  upgrade/mimic-x/parallel/mon_election/classic
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552288/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552288/
sentry: https://sentry.ceph.com/sentry/sepia/?query=fe5ca9be62654034b26b7315a0ac8b0f

    list index out of range

[5552289]  upgrade/mimic-x-singleton/mon_election/classic
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552289/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552289/
sentry: https://sentry.ceph.com/sentry/sepia/?query=52d613b7f128414ab1a95f81caa92639

    list index out of range

[5552290]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:19:51
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552290/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552290/
sentry: https://sentry.ceph.com/sentry/sepia/?query=5694fb23962b43b1aa47ba5318503f0b

    timeout expired in wait_until_healthy

[5552291]  upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   04:48:18
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552291/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552291/

    "2020-10-24T01:30:52.456445+0000 mon.c (mon.0) 15 : cluster [WRN] Health
    check failed: There are daemons running an older version of ceph
    (DAEMON_OLD_VERSION)" in cluster log

[5552292]  upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:19:26
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552292/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552292/
sentry: https://sentry.ceph.com/sentry/sepia/?query=313d506f1da947678de368ed51e55649

    timeout expired in wait_until_healthy

[5552293]  upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:18:51
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552293/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552293/
sentry: https://sentry.ceph.com/sentry/sepia/?query=ec8bcaf2243f44499d072420585692bc

    timeout expired in wait_until_healthy

[5552294]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:22:29
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552294/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552294/
sentry: https://sentry.ceph.com/sentry/sepia/?query=d4e694e618274bb38c2bf46d667ec389

    timeout expired in wait_until_healthy

[5552295]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:20:03
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552295/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552295/
sentry: https://sentry.ceph.com/sentry/sepia/?query=ff12a84854c34b6ebf11898bc382db66

    timeout expired in wait_until_healthy

[5552296]  upgrade/cephfs/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04}
-----------------------------------------------------------------
time:   01:11:24
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552296/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552296/
sentry: https://sentry.ceph.com/sentry/sepia/?query=d9c667e696f247ae87a8d328641237a7

    timeout expired in wait_until_healthy

[5552297]  upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   03:04:39
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552297/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552297/

    "2020-10-24T00:38:35.352444+0000 mon.c (mon.0) 25 : cluster [WRN] Health
    check failed: There are daemons running an older version of ceph
    (DAEMON_OLD_VERSION)" in cluster log

[5552298]  upgrade/mimic-x/stress-split/mon_election/classic
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552298/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552298/
sentry: https://sentry.ceph.com/sentry/sepia/?query=5612a421c66243789ac154c0f6a7bc51

    list index out of range

[5552299]  upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:19:41
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552299/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552299/
sentry: https://sentry.ceph.com/sentry/sepia/?query=e8c579080e2241bbad2c6b824a69b9e8

    timeout expired in wait_until_healthy

[5552300]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:18:24
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552300/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552300/
sentry: https://sentry.ceph.com/sentry/sepia/?query=b241516fa67f4ac08a14c14d577fbea5

    timeout expired in wait_until_healthy

[5552301]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:21:28
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552301/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552301/
sentry: https://sentry.ceph.com/sentry/sepia/?query=0d919747d73e4a379cbe79124900b299

    timeout expired in wait_until_healthy

[5552302]  upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:19:09
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552302/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552302/
sentry: https://sentry.ceph.com/sentry/sepia/?query=e3967aaa1fd74c8393e06e6c8e778df2

    timeout expired in wait_until_healthy

[5552303]  upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:13:28
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552303/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552303/
sentry: https://sentry.ceph.com/sentry/sepia/?query=d29219a62e514337acec632c14728f17

    timeout expired in wait_until_healthy

[5552304]  upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:18:55
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552304/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552304/
sentry: https://sentry.ceph.com/sentry/sepia/?query=f734badc38334fae939d5c1c4a2b2f06

    timeout expired in wait_until_healthy

[5552305]  upgrade/mimic-x/stress-split-erasure-code/mon_election/classic
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552305/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552305/
sentry: https://sentry.ceph.com/sentry/sepia/?query=6da227e862d04b0abed81061001e788b

    list index out of range

[5552306]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:19:10
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552306/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552306/
sentry: https://sentry.ceph.com/sentry/sepia/?query=28803fdb504a4602985dab08cda3f8c7

    timeout expired in wait_until_healthy

[5552307]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:20:10
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552307/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552307/
sentry: https://sentry.ceph.com/sentry/sepia/?query=00e49a354dfc4500ab14e61e2a23d298

    timeout expired in wait_until_healthy

[5552308]  upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:18:17
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552308/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552308/
sentry: https://sentry.ceph.com/sentry/sepia/?query=60d2ed26c0c64c239b1979dc7b71768b

    timeout expired in wait_until_healthy

[5552309]  upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
-----------------------------------------------------------------
time:   00:23:34
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552309/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552309/
sentry: https://sentry.ceph.com/sentry/sepia/?query=3a8973bea27242dd901507f68e437ca4

    timeout expired in wait_until_healthy

[5552310]  upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:18:24
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552310/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552310/
sentry: https://sentry.ceph.com/sentry/sepia/?query=5598bf30a64248fd8bdbc1eda685dda4

    timeout expired in wait_until_healthy

[5552311]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:18:47
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552311/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552311/
sentry: https://sentry.ceph.com/sentry/sepia/?query=3e646459d93a4b1b806af3711e448f0c

    timeout expired in wait_until_healthy

[5552312]  upgrade/mimic-x/parallel/mon_election/connectivity
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552312/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552312/
sentry: https://sentry.ceph.com/sentry/sepia/?query=51bea7c18991459e8fb0ebc3afccf356

    list index out of range

[5552313]  upgrade/mimic-x-singleton/mon_election/connectivity
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552313/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552313/
sentry: https://sentry.ceph.com/sentry/sepia/?query=e9d268eaab0641cc9ae9ee127411397e

    list index out of range

[5552314]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:20:26
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552314/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552314/
sentry: https://sentry.ceph.com/sentry/sepia/?query=6d454a731f3641aba581d5790cf19fb2

    timeout expired in wait_until_healthy

[5552315]  upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   04:54:23
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552315/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552315/

    "2020-10-23T22:44:25.948655+0000 mon.b (mon.0) 7 : cluster [WRN] Health
    check failed: There are daemons running an older version of ceph
    (DAEMON_OLD_VERSION)" in cluster log

[5552316]  upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   05:13:19
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552316/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552316/

    "2020-10-24T01:24:52.797582+0000 mon.c (mon.0) 22 : cluster [WRN] Health
    check failed: There are daemons running an older version of ceph
    (DAEMON_OLD_VERSION)" in cluster log

[5552317]  upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   03:09:03
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552317/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552317/

    "2020-10-23T23:57:37.377259+0000 mon.c (mon.0) 20 : cluster [WRN] Health
    check failed: There are daemons running an older version of ceph
    (DAEMON_OLD_VERSION)" in cluster log

[5552318]  upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:23:57
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552318/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552318/
sentry: https://sentry.ceph.com/sentry/sepia/?query=96a0525c47414243a8d19d38591aaf0f

    timeout expired in wait_until_healthy

[5552319]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:19:09
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552319/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552319/
sentry: https://sentry.ceph.com/sentry/sepia/?query=c233d38a726c495faa473ec1586dc75d

    timeout expired in wait_until_healthy

[5552320]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:20:17
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552320/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552320/
sentry: https://sentry.ceph.com/sentry/sepia/?query=d48e2ac932f647408c3d405c0c35c0bd

    timeout expired in wait_until_healthy

[5552321]  upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   03:20:51
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552321/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552321/

    "2020-10-24T02:12:20.860763+0000 mon.c (mon.0) 34 : cluster [WRN] Health
    check failed: There are daemons running an older version of ceph
    (DAEMON_OLD_VERSION)" in cluster log

[5552322]  upgrade/mimic-x/stress-split/mon_election/connectivity
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552322/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552322/
sentry: https://sentry.ceph.com/sentry/sepia/?query=3f68455dec244561a51a25f4c19de4f9

    list index out of range

[5552323]  upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   01:53:05
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552323/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552323/

    "2020-10-24T02:00:00.021554+0000 mon.c (mon.0) 29 : cluster [WRN] Health
    check failed: There are daemons running an older version of ceph
    (DAEMON_OLD_VERSION)" in cluster log

[5552324]  upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:14:32
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552324/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552324/
sentry: https://sentry.ceph.com/sentry/sepia/?query=a213e717939a4e8993511f88613a6af3

    timeout expired in wait_until_healthy

[5552325]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:22:17
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552325/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552325/
sentry: https://sentry.ceph.com/sentry/sepia/?query=099865ede46d44aa963dd3b284651d03

    timeout expired in wait_until_healthy

[5552326]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:21:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552326/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552326/
sentry: https://sentry.ceph.com/sentry/sepia/?query=e7e8d0d4041d45e99a2ca01fee673548

    timeout expired in wait_until_healthy

[5552327]  upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:18:36
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552327/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552327/
sentry: https://sentry.ceph.com/sentry/sepia/?query=ae9373e4d6334ee88bb833748d3d30e4

    timeout expired in wait_until_healthy

[5552328]  upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:19:13
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552328/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552328/
sentry: https://sentry.ceph.com/sentry/sepia/?query=a8fd83b4e1c9410b8f6ce9eb9f92e6e3

    timeout expired in wait_until_healthy

[5552329]  upgrade/mimic-x/stress-split-erasure-code/mon_election/connectivity
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552329/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552329/
sentry: https://sentry.ceph.com/sentry/sepia/?query=6a52ec00582c4b39abc3a012e239e5f3

    list index out of range

[5552330]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:19:33
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552330/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552330/
sentry: https://sentry.ceph.com/sentry/sepia/?query=f36d66a6026346c989937beef9198fe3

    timeout expired in wait_until_healthy

[5552331]  upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
-----------------------------------------------------------------
time:   00:24:21
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552331/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552331/
sentry: https://sentry.ceph.com/sentry/sepia/?query=b39db852e29244edbbaea55989b1e05e

    timeout expired in wait_until_healthy

[5552332]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:22:26
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552332/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552332/
sentry: https://sentry.ceph.com/sentry/sepia/?query=6719926f7b2941119acdd770ea5259c6

    timeout expired in wait_until_healthy

[5552333]  upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   03:16:03
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552333/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552333/

    "2020-10-24T01:21:07.302204+0000 mon.c (mon.0) 24 : cluster [WRN] Health
    check failed: There are daemons running an older version of ceph
    (DAEMON_OLD_VERSION)" in cluster log

[5552334]  upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:19:00
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552334/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552334/
sentry: https://sentry.ceph.com/sentry/sepia/?query=f76a94fa381f45ec9a0f9e71326ed67c

    timeout expired in wait_until_healthy

[5552335]  upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:18:39
info:   http://pulpito.ceph.com/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552335/
log:    http://qa-proxy.ceph.com/teuthology/dzafman-2020-10-23_22:00:18-upgrade-wip-zafman-46264-2-distro-basic-smithi/5552335/
sentry: https://sentry.ceph.com/sentry/sepia/?query=46e50894bef64b3797bc93b38e3a2db7

    timeout expired in wait_until_healthy