2023-04-09T10:32:37.851 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-09T10:32:37.852 INFO:teuthology.results:seed: '1820' 2023-04-09T10:37:57.625 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-09T10:37:57.628 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-09T15:32:20.791 INFO:teuthology.results:Tests finished! gathering results... 2023-04-09T15:32:21.394 INFO:scrape:Found 6 jobs 2023-04-09T15:32:21.399 INFO:scrape:Found 3 distinct failure reasons 2023-04-09T15:32:21.400 INFO:scrape:Failure: "2023-04-09T13:00:00.000175+0000 mon.a (mon.0) 1802 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 3 pgs inactive, 4 pgs peering; Degraded data redundancy: 37/302 objects degraded (12.252%), 9 pgs degraded" in cluster log 2023-04-09T15:32:21.400 INFO:scrape:1 jobs: ['7236906'] 2023-04-09T15:32:21.401 INFO:scrape:suites: ['supported-all-distro/centos_8}', 'upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade'] 2023-04-09T15:32:21.401 INFO:scrape: 2023-04-09T15:32:21.401 INFO:scrape:Failure: "2023-04-09T11:13:02.315075+0000 mon.a (mon.0) 384 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log 2023-04-09T15:32:21.402 INFO:scrape:1 jobs: ['7236909'] 2023-04-09T15:32:21.402 INFO:scrape:suites: ['supported-all-distro/ubuntu_latest}', 'upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade'] 2023-04-09T15:32:21.402 INFO:scrape: 2023-04-09T15:32:21.402 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2023-04-09T15:32:21.403 INFO:scrape:1 jobs: ['7236911'] 2023-04-09T15:32:21.403 INFO:scrape:suites: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack'] 2023-04-09T15:32:21.403 INFO:scrape: 2023-04-09T15:32:21.404 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 3 Fail ================================================================= [7236906] upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8} ----------------------------------------------------------------- time: 03:31:49 info: http://pulpito.ceph.com/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/7236906/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/7236906/ "2023-04-09T13:00:00.000175+0000 mon.a (mon.0) 1802 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 3 pgs inactive, 4 pgs peering; Degraded data redundancy: 37/302 objects degraded (12.252%), 9 pgs degraded" in cluster log [7236909] upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} ----------------------------------------------------------------- time: 01:21:49 info: http://pulpito.ceph.com/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/7236909/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/7236909/ "2023-04-09T11:13:02.315075+0000 mon.a (mon.0) 384 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7236911] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 01:37:18 info: http://pulpito.ceph.com/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/7236911/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/7236911/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb7f95f3aabd40058195e9ff75e84b07 Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' Pass ================================================================= [7236907] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} time: 04:38:51 info: http://pulpito.ceph.com/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/7236907/ [7236908] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health} time: 03:24:29 info: http://pulpito.ceph.com/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/7236908/ [7236910] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health} time: 04:45:06 info: http://pulpito.ceph.com/teuthology-2023-04-09_01:35:03-upgrade:quincy-p2p-quincy-distro-default-smithi/7236910/