2022-04-27T17:54:51.130 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-04-27T17:54:51.131 INFO:teuthology.results:subset: '111/120000' 2022-04-27T17:54:51.131 INFO:teuthology.results:seed: '3097' 2022-04-27T18:10:49.292 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-04-27T18:10:50.165 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-28T00:51:19.220 INFO:teuthology.results:Tests finished! gathering results... 2022-04-28T00:51:20.501 INFO:scrape:Found 11 jobs 2022-04-28T00:51:20.637 INFO:scrape:Found 3 distinct failure reasons 2022-04-28T00:51:20.637 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2022-04-28T00:51:20.637 INFO:scrape:2 jobs: ['6808897', '6808903'] 2022-04-28T00:51:20.638 INFO:scrape:suites intersection: ['1-ceph-install/nautilus', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack'] 2022-04-28T00:51:20.638 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack'] 2022-04-28T00:51:20.638 INFO:scrape: 2022-04-28T00:51:20.638 INFO:scrape:Dead: b'2022-04-28T00:41:03.852 DEBUG:teuthology.exit:Finished running handlers' 2022-04-28T00:51:20.638 INFO:scrape:4 jobs: ['6808901', '6808898', '6808904', '6808907'] 2022-04-28T00:51:20.639 INFO:scrape:suites intersection: ['1-nautilus-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-pacific', '7-final-workload', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack'] 2022-04-28T00:51:20.639 INFO:scrape:suites union: ['1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-pacific', '7-final-workload', 'mon_election/classic', 'mon_election/connectivity', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack'] 2022-04-28T00:51:20.639 INFO:scrape: 2022-04-28T00:51:20.639 INFO:scrape:Failure: timeout expired in wait_for_all_osds_up 2022-04-28T00:51:20.639 INFO:scrape:1 jobs: ['6808902'] 2022-04-28T00:51:20.640 INFO:scrape:suites: ['1-ceph-install/nautilus', '1.1-pg-log-overrides/short_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-mon-osd-mds', '4-pacific', '5-final-workload/{blogbench', 'ec-rados-default', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_18.04}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack'] 2022-04-28T00:51:20.640 INFO:scrape: 2022-04-28T00:51:20.641 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/ failed: 3 dead: 4 running: 0 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [6808897] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:17:13 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808897/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808897/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64fc9f951db64f33b23ed0525b3f53a4 Command failed on smithi005 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [6808902] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:49:37 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808902/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808902/ sentry: https://sentry.ceph.com/organizations/ceph/?query=acf7c19693a84db9b1a25ba4417e588e timeout expired in wait_for_all_osds_up [6808903] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:11:02 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808903/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808903/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54f6f797c9774d0aa14fc35b0d7591ea Command failed on smithi018 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' Dead ================================================================= [6808898] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808898/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808898/ hit max job timeout [6808901] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808901/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808901/ hit max job timeout [6808904] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808904/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808904/ hit max job timeout [6808907] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808907/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808907/ hit max job timeout Pass ================================================================= [6808899] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} time: 02:11:59 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808899/ [6808900] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 03:07:10 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808900/ [6808905] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} time: 02:10:15 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808905/ [6808906] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:45:54 info: http://pulpito.ceph.com/yuriw-2022-04-27_14:24:05-upgrade:nautilus-x-pacific-distro-default-smithi/6808906/