2021-10-12T18:29:31.468 INFO:root:teuthology version: 1.1.0-6f40c4ca 2021-10-12T18:29:31.469 INFO:teuthology.results:subset: '8/10' 2021-10-12T18:29:31.469 INFO:teuthology.results:seed: '2150' 2021-10-12T19:36:30.482 INFO:root:teuthology version: 1.1.0-6f40c4ca 2021-10-12T19:36:31.746 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-10-13T00:01:52.074 INFO:teuthology.results:Tests finished! gathering results... 2021-10-13T00:01:55.787 INFO:scrape:Found 25 jobs 2021-10-13T00:01:56.661 INFO:scrape:Found 1 distinct failure reasons 2021-10-13T00:01:56.661 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2021-10-13T00:01:56.662 INFO:scrape:2 jobs: ['6433913', '6433896'] 2021-10-13T00:01:56.662 INFO:scrape:suites intersection: ['1.1-pg-log-overrides/normal_pg_log', 'start}'] 2021-10-13T00:01:56.662 INFO:scrape:suites union: ['1-ceph-install/octopus', '1-nautilus-install/octopus', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '2-workload/{rgw_ragweed_prepare}', '3-thrash/default', '3-upgrade-sequence/upgrade-mon-osd-mds', '3.1-objectstore/bluestore-bitmap', '4-ec-workload/{rados-ec-workload', '4-pacific', '5-final-workload/{rgw', '5-finish-upgrade', '6-pacific', '7-final-workload', 'centos_latest}', 'mon_election/classic', 'rbd-ec-workload}', 'rgw_ragweed_check}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack'] 2021-10-13T00:01:56.662 INFO:scrape: 2021-10-13T00:01:56.664 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 23 Fail ================================================================= [6433896] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:23:55 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433896/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433896/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a8c08fde75e45ca9ae055d7595f84fe Command failed on smithi137 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [6433913] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} ----------------------------------------------------------------- time: 00:24:27 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433913/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433913/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52a4af8fd074469bba09a3732969ed15 Command failed on smithi178 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' Pass ================================================================= [6433892] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:16 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433892/ [6433893] upgrade:octopus-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 01:58:28 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433893/ [6433894] upgrade:octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:17 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433894/ [6433895] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:20:36 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433895/ [6433897] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 04:42:20 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433897/ [6433898] upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:54:15 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433898/ [6433899] upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:39 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433899/ [6433900] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:17:51 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433900/ [6433901] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:26:41 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433901/ [6433902] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 00:51:45 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433902/ [6433903] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:35:29 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433903/ [6433904] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:17 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433904/ [6433905] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:28:13 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433905/ [6433906] upgrade:octopus-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:00:20 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433906/ [6433907] upgrade:octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:12 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433907/ [6433908] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:20:40 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433908/ [6433909] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 00:48:34 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433909/ [6433910] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 04:40:49 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433910/ [6433911] upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:12:08 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433911/ [6433912] upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:06 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433912/ [6433914] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:10:39 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433914/ [6433915] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 00:49:59 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433915/ [6433916] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:38:43 info: http://pulpito.ceph.com/teuthology-2021-10-12_13:10:21-upgrade:octopus-x-pacific-distro-basic-smithi/6433916/