2021-11-13T14:29:47.675 INFO:root:teuthology version: 1.1.0-d4737010 2021-11-13T14:29:47.676 INFO:teuthology.results:subset: '111/120000' 2021-11-13T14:29:47.677 INFO:teuthology.results:seed: '3169' 2021-11-13T20:15:02.175 INFO:root:teuthology version: 1.1.0-d4737010 2021-11-13T20:15:02.214 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-11-14T08:09:59.413 INFO:teuthology.results:Tests finished! gathering results... 2021-11-14T08:10:09.240 INFO:scrape:Found 87 jobs 2021-11-14T08:10:10.225 INFO:scrape:Found 4 distinct failure reasons 2021-11-14T08:10:10.226 INFO:scrape:Failure: Command failed on smithi172 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.15-32-gb0aaed0f-1focal cephadm=15.2.15-32-gb0aaed0f-1focal ceph-mds=15.2.15-32-gb0aaed0f-1focal ceph-mgr=15.2.15-32-gb0aaed0f-1focal ceph-common=15.2.15-32-gb0aaed0f-1focal ceph-fuse=15.2.15-32-gb0aaed0f-1focal ceph-test=15.2.15-32-gb0aaed0f-1focal ceph-volume=15.2.15-32-gb0aaed0f-1focal radosgw=15.2.15-32-gb0aaed0f-1focal python3-rados=15.2.15-32-gb0aaed0f-1focal python3-rgw=15.2.15-32-gb0aaed0f-1focal python3-cephfs=15.2.15-32-gb0aaed0f-1focal python3-rbd=15.2.15-32-gb0aaed0f-1focal libcephfs2=15.2.15-32-gb0aaed0f-1focal libcephfs-dev=15.2.15-32-gb0aaed0f-1focal librados2=15.2.15-32-gb0aaed0f-1focal librbd1=15.2.15-32-gb0aaed0f-1focal rbd-fuse=15.2.15-32-gb0aaed0f-1focal' 2021-11-14T08:10:10.226 INFO:scrape:4 jobs: ['6501129', '6501173', '6501195', '6501152'] 2021-11-14T08:10:10.227 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/octopus-x/rgw-multisite/{clusters'] 2021-11-14T08:10:10.227 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/octopus-x/rgw-multisite/{clusters', 'upgrade/primary}', 'upgrade/secondary}'] 2021-11-14T08:10:10.228 INFO:scrape: 2021-11-14T08:10:10.228 INFO:scrape:Dead: b'2021-11-14T03:27:39.313 DEBUG:teuthology.exit:Finished running handlers' 2021-11-14T08:10:10.229 INFO:scrape:4 jobs: ['6501159', '6501135', '6501183', '6501199'] 2021-11-14T08:10:10.229 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '3-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'tasks/{0-v16.2.4', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-11-14T08:10:10.230 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-upgrade-with-workload', '3-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'tasks/{0-v16.2.4', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-11-14T08:10:10.230 INFO:scrape: 2021-11-14T08:10:10.231 INFO:scrape:Failure: Command failed on smithi090 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' 2021-11-14T08:10:10.231 INFO:scrape:7 jobs: ['6501154', '6501197', '6501187', '6501164', '6501142', '6501177', '6501133'] 2021-11-14T08:10:10.232 INFO:scrape:suites intersection: ['1-ceph-install/octopus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2021-11-14T08:10:10.233 INFO:scrape:suites union: ['1-ceph-install/octopus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2021-11-14T08:10:10.233 INFO:scrape: 2021-11-14T08:10:10.234 INFO:scrape:Failure: wait_for_recovery: failed before timeout expired 2021-11-14T08:10:10.234 INFO:scrape:1 jobs: ['6501209'] 2021-11-14T08:10:10.234 INFO:scrape:suites: ['1-ceph-install/octopus', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'mon_election/classic', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2021-11-14T08:10:10.235 INFO:scrape: 2021-11-14T08:10:10.241 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/ failed: 12 dead: 4 running: 0 waiting: 0 queued: 0 passed: 71 Fail ================================================================= [6501129] upgrade/octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:03:43 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501129/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501129/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49d69f52445841f0aa978493095790e2 Command failed on smithi172 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.15-32-gb0aaed0f-1focal cephadm=15.2.15-32-gb0aaed0f-1focal ceph- mds=15.2.15-32-gb0aaed0f-1focal ceph-mgr=15.2.15-32-gb0aaed0f-1focal ceph- common=15.2.15-32-gb0aaed0f-1focal ceph-fuse=15.2.15-32-gb0aaed0f-1focal ceph-test=15.2.15-32-gb0aaed0f-1focal ceph- volume=15.2.15-32-gb0aaed0f-1focal radosgw=15.2.15-32-gb0aaed0f-1focal python3-rados=15.2.15-32-gb0aaed0f-1focal python3-rgw=15.2.15-32-gb0aaed0f-1focal python3-cephfs=15.2.15-32-gb0aaed0f-1focal python3-rbd=15.2.15-32-gb0aaed0f-1focal libcephfs2=15.2.15-32-gb0aaed0f-1focal libcephfs- dev=15.2.15-32-gb0aaed0f-1focal librados2=15.2.15-32-gb0aaed0f-1focal librbd1=15.2.15-32-gb0aaed0f-1focal rbd-fuse=15.2.15-32-gb0aaed0f-1focal' [6501133] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 04:20:41 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501133/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501133/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9579b250b9d944de8791c6417030d1ab Command failed on smithi155 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6501142] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:46:59 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501142/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501142/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73fb13cfd25148b9a792d103aa4d59ec Command failed on smithi174 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6501152] upgrade/octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:03:44 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501152/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501152/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ef93ba592cf4e95a1005d629f9f248e Command failed on smithi157 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.15-32-gb0aaed0f-1focal cephadm=15.2.15-32-gb0aaed0f-1focal ceph- mds=15.2.15-32-gb0aaed0f-1focal ceph-mgr=15.2.15-32-gb0aaed0f-1focal ceph- common=15.2.15-32-gb0aaed0f-1focal ceph-fuse=15.2.15-32-gb0aaed0f-1focal ceph-test=15.2.15-32-gb0aaed0f-1focal ceph- volume=15.2.15-32-gb0aaed0f-1focal radosgw=15.2.15-32-gb0aaed0f-1focal python3-rados=15.2.15-32-gb0aaed0f-1focal python3-rgw=15.2.15-32-gb0aaed0f-1focal python3-cephfs=15.2.15-32-gb0aaed0f-1focal python3-rbd=15.2.15-32-gb0aaed0f-1focal libcephfs2=15.2.15-32-gb0aaed0f-1focal libcephfs- dev=15.2.15-32-gb0aaed0f-1focal librados2=15.2.15-32-gb0aaed0f-1focal librbd1=15.2.15-32-gb0aaed0f-1focal rbd-fuse=15.2.15-32-gb0aaed0f-1focal' [6501154] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 04:03:51 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501154/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501154/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9717be74dcdb4f2384a2b48e6ee106d5 Command failed on smithi090 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6501164] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:45:39 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501164/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501164/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4166ffe7b6134c11b20fb2c0430fd77f Command failed on smithi187 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6501173] upgrade/octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:03:39 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501173/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501173/ sentry: https://sentry.ceph.com/organizations/ceph/?query=928131a35a5646149304ecaa7591da28 Command failed on smithi152 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.15-32-gb0aaed0f-1focal cephadm=15.2.15-32-gb0aaed0f-1focal ceph- mds=15.2.15-32-gb0aaed0f-1focal ceph-mgr=15.2.15-32-gb0aaed0f-1focal ceph- common=15.2.15-32-gb0aaed0f-1focal ceph-fuse=15.2.15-32-gb0aaed0f-1focal ceph-test=15.2.15-32-gb0aaed0f-1focal ceph- volume=15.2.15-32-gb0aaed0f-1focal radosgw=15.2.15-32-gb0aaed0f-1focal python3-rados=15.2.15-32-gb0aaed0f-1focal python3-rgw=15.2.15-32-gb0aaed0f-1focal python3-cephfs=15.2.15-32-gb0aaed0f-1focal python3-rbd=15.2.15-32-gb0aaed0f-1focal libcephfs2=15.2.15-32-gb0aaed0f-1focal libcephfs- dev=15.2.15-32-gb0aaed0f-1focal librados2=15.2.15-32-gb0aaed0f-1focal librbd1=15.2.15-32-gb0aaed0f-1focal rbd-fuse=15.2.15-32-gb0aaed0f-1focal' [6501177] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 04:09:04 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501177/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501177/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b49265929cf404592a152f753ec5ddd Command failed on smithi120 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6501187] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 02:15:22 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501187/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501187/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5e0f07408cd4382a491bdb0388a2f23 Command failed on smithi165 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6501195] upgrade/octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:03:37 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501195/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f41b392bb8fa4fd294544331db7f73ec Command failed on smithi083 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.15-32-gb0aaed0f-1focal cephadm=15.2.15-32-gb0aaed0f-1focal ceph- mds=15.2.15-32-gb0aaed0f-1focal ceph-mgr=15.2.15-32-gb0aaed0f-1focal ceph- common=15.2.15-32-gb0aaed0f-1focal ceph-fuse=15.2.15-32-gb0aaed0f-1focal ceph-test=15.2.15-32-gb0aaed0f-1focal ceph- volume=15.2.15-32-gb0aaed0f-1focal radosgw=15.2.15-32-gb0aaed0f-1focal python3-rados=15.2.15-32-gb0aaed0f-1focal python3-rgw=15.2.15-32-gb0aaed0f-1focal python3-cephfs=15.2.15-32-gb0aaed0f-1focal python3-rbd=15.2.15-32-gb0aaed0f-1focal libcephfs2=15.2.15-32-gb0aaed0f-1focal libcephfs- dev=15.2.15-32-gb0aaed0f-1focal librados2=15.2.15-32-gb0aaed0f-1focal librbd1=15.2.15-32-gb0aaed0f-1focal rbd-fuse=15.2.15-32-gb0aaed0f-1focal' [6501197] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:54:28 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501197/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501197/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2d1075f98b04cdb961b3ec5cf4069d7 Command failed on smithi099 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6501209] upgrade/octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:59:15 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501209/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4de25cb7bbaa4da982e575f65619c441 wait_for_recovery: failed before timeout expired Dead ================================================================= [6501135] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501135/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501135/ hit max job timeout [6501159] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501159/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501159/ hit max job timeout [6501183] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501183/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501183/ hit max job timeout [6501199] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501199/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501199/ hit max job timeout Pass ================================================================= [6501125] upgrade/octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:35 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501125/ [6501126] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:54 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501126/ [6501127] upgrade/octopus-x/parallel/{0-distro/centos_8.2_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:51 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501127/ [6501128] upgrade/pacific-x/parallel/{0-distro$/{rhel_8.4_container_tools_rhel8} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:31:02 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501128/ [6501130] upgrade/octopus-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:21:48 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501130/ [6501131] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 00:48:40 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501131/ [6501132] upgrade/pacific-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:20:12 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501132/ [6501134] upgrade/octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:18 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501134/ [6501136] upgrade/octopus-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:50:04 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501136/ [6501137] upgrade/pacific-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:52:39 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501137/ [6501138] upgrade/octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:52 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501138/ [6501139] upgrade/octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:18:23 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501139/ [6501140] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 00:48:35 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501140/ [6501141] upgrade/pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:06:01 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501141/ [6501143] upgrade/cephfs/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:10:11 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501143/ [6501144] upgrade/octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:00:22 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501144/ [6501145] upgrade/octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:17:23 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501145/ [6501146] upgrade/pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:04:54 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501146/ [6501147] upgrade/octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:07 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501147/ [6501148] upgrade/octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:03:29 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501148/ [6501149] upgrade/octopus-x/parallel/{0-distro/centos_8.2_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:50 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501149/ [6501150] upgrade/pacific-x/parallel/{0-distro$/{rhel_8.4_container_tools_3.0} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:30:14 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501150/ [6501151] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:18:57 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501151/ [6501153] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 00:46:24 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501153/ [6501155] upgrade/octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:30:55 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501155/ [6501156] upgrade/pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:22:21 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501156/ [6501157] upgrade/octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:05 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501157/ [6501158] upgrade/octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:57 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501158/ [6501160] upgrade/octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:12:05 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501160/ [6501161] upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:23:48 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501161/ [6501162] upgrade/octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:17:05 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501162/ [6501163] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 00:49:31 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501163/ [6501165] upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:48:05 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501165/ [6501166] upgrade/octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:37 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501166/ [6501167] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:24:17 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501167/ [6501168] upgrade/octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:31:40 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501168/ [6501169] upgrade/octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:14 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501169/ [6501170] upgrade/pacific-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:50:35 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501170/ [6501171] upgrade/octopus-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:57:32 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501171/ [6501172] upgrade/octopus-x/parallel/{0-distro/centos_8.2_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:13 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501172/ [6501174] upgrade/pacific-x/parallel/{0-distro$/{centos_8.2_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:18:33 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501174/ [6501175] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:16 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501175/ [6501176] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 00:49:30 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501176/ [6501178] upgrade/octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:29 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501178/ [6501179] upgrade/pacific-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:13:51 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501179/ [6501180] upgrade/octopus-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:19:57 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501180/ [6501181] upgrade/octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:37 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501181/ [6501182] upgrade/octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:51 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501182/ [6501184] upgrade/octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:52:42 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501184/ [6501185] upgrade/pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:51:02 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501185/ [6501186] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 00:47:20 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501186/ [6501188] upgrade/octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:47 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501188/ [6501189] upgrade/pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:33:43 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501189/ [6501190] upgrade/octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:30:35 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501190/ [6501191] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:19 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501191/ [6501192] upgrade/octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:23 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501192/ [6501193] upgrade/octopus-x/parallel/{0-distro/centos_8.2_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:07 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501193/ [6501194] upgrade/pacific-x/parallel/{0-distro$/{centos_8.2_container_tools_3.0} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:18:56 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501194/ [6501196] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 00:46:02 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501196/ [6501198] upgrade/pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:30:48 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501198/ [6501200] upgrade/octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:25:45 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501200/ [6501201] upgrade/octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:18 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501201/ [6501202] upgrade/octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:39 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501202/ [6501203] upgrade/pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:33:03 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501203/ [6501204] upgrade/octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:41:15 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501204/ [6501205] upgrade/octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:25 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501205/ [6501206] upgrade/octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 00:51:28 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501206/ [6501207] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:22:44 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501207/ [6501208] upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:10:29 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501208/ [6501210] upgrade/octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:00:46 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501210/ [6501211] upgrade/octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:18:09 info: http://pulpito.ceph.com/yuriw-2021-11-13_14:29:01-upgrade-wip-yuriw-master-11.12.21-distro-basic-smithi/6501211/