2022-04-22T14:22:51.468 INFO:root:teuthology version: 1.1.1.dev313+g1b11219b 2022-04-22T14:22:51.469 INFO:teuthology.results:subset: '8/10' 2022-04-22T14:22:51.469 INFO:teuthology.results:seed: '1770' 2022-04-22T16:21:13.549 INFO:root:teuthology version: 1.1.1.dev313+g1b11219b 2022-04-22T16:21:13.571 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-22T20:38:32.842 INFO:teuthology.results:Tests finished! gathering results... 2022-04-22T20:38:35.920 INFO:scrape:Found 32 jobs 2022-04-22T20:38:36.598 INFO:scrape:Found 2 distinct failure reasons 2022-04-22T20:38:36.598 INFO:scrape:Failure: No module named 'nose' 2022-04-22T20:38:36.598 INFO:scrape:2 jobs: ['6800603', '6800619'] 2022-04-22T20:38:36.599 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-04-22T20:38:36.599 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-04-22T20:38:36.599 INFO:scrape: 2022-04-22T20:38:36.599 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi178.front.sepia.ceph.com: ['type=AVC msg=audit(1650638554.924:199): avc: denied { node_bind } for pid=2109 comm="ping" saddr=172.21.15.178 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2022-04-22T20:38:36.599 INFO:scrape:1 jobs: ['6800624'] 2022-04-22T20:38:36.599 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'workload/{ec-rados-default'] 2022-04-22T20:38:36.600 INFO:scrape: 2022-04-22T20:38:36.602 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 29 Fail ================================================================= [6800603] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:06:33 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800603/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800603/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57e00f9aa35d4ea69de4720d877bd1bd No module named 'nose' [6800619] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:06:31 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800619/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800619/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ebfccbf167142e9b05dde08b9291b7b No module named 'nose' [6800624] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 01:01:25 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800624/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800624/ SELinux denials found on ubuntu@smithi178.front.sepia.ceph.com: ['type=AVC msg=audit(1650638554.924:199): avc: denied { node_bind } for pid=2109 comm="ping" saddr=172.21.15.178 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] Pass ================================================================= [6800602] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:58 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800602/ [6800604] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:05:50 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800604/ [6800605] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 03:48:30 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800605/ [6800606] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:31:32 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800606/ [6800607] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:13:15 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800607/ [6800608] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:59 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800608/ [6800609] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:22:18 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800609/ [6800610] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:13:53 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800610/ [6800611] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:26:07 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800611/ [6800612] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 01:42:57 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800612/ [6800613] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:25:50 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800613/ [6800614] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:13:28 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800614/ [6800615] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:41:47 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800615/ [6800616] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:30 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800616/ [6800617] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:18:24 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800617/ [6800618] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:29 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800618/ [6800620] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:12:14 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800620/ [6800621] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:16:31 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800621/ [6800622] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 05:56:14 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800622/ [6800623] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:04:16 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800623/ [6800625] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:26:33 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800625/ [6800626] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:15:01 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800626/ [6800627] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 01:57:32 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800627/ [6800628] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:36:14 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800628/ [6800629] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:35:06 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800629/ [6800630] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:11:15 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800630/ [6800631] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:45:16 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800631/ [6800632] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:57 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800632/ [6800633] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:21:18 info: http://pulpito.ceph.com/teuthology-2022-04-22_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6800633/