2023-04-14T14:23:15.952 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-14T14:23:15.953 INFO:teuthology.results:subset: '111/120000' 2023-04-14T14:23:15.953 INFO:teuthology.results:seed: '7425' 2023-04-15T17:10:30.760 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-15T17:10:30.791 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-16T05:11:20.554 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-04-16T05:11:23.296 INFO:scrape:Found 27 jobs 2023-04-16T05:11:25.549 INFO:scrape:Found 6 distinct failure reasons 2023-04-16T05:11:25.550 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-04-16T05:11:25.550 INFO:scrape:2 jobs: ['7241195', '7241181'] 2023-04-16T05:11:25.551 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-04-16T05:11:25.551 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-04-16T05:11:25.551 INFO:scrape: 2023-04-16T05:11:25.552 INFO:scrape:Dead: b'2023-04-15T02:31:51.105 DEBUG:teuthology.exit:Finished running handlers' 2023-04-16T05:11:25.552 INFO:scrape:3 jobs: ['7241180', '7241185', '7241191'] 2023-04-16T05:11:25.552 INFO:scrape:suites intersection: [] 2023-04-16T05:11:25.553 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'workload/{ec-rados-default'] 2023-04-16T05:11:25.553 INFO:scrape: 2023-04-16T05:11:25.553 INFO:scrape:Failure: {'smithi081.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.198)", 'rc': 1, 'results': []}, 'smithi044.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 8.43.85.73)", 'rc': 1, 'results': []}} 2023-04-16T05:11:25.554 INFO:scrape:7 jobs: ['7241193', '7241202', '7241190', '7241176', '7241173', '7241177', '7241186'] 2023-04-16T05:11:25.554 INFO:scrape:suites intersection: [] 2023-04-16T05:11:25.554 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8', 'workload/{ec-rados-default'] 2023-04-16T05:11:25.555 INFO:scrape: 2023-04-16T05:11:25.555 INFO:scrape:Dead: b'2023-04-16T05:00:03.264 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Apr 16 05:00:02 smithi174 ceph-1f07a8d6-dbc1-11ed-9aff-001a4aab830c-prometheus-a[170590]: ts=2023-04-16T05:00:02.887Z caller=db.go:1287 level=info component=tsdb msg="Deleting obsolete block" block=01GY3S6ZBSDNDGDQWB11AWTYC7' 2023-04-16T05:11:25.555 INFO:scrape:1 jobs: ['7241200'] 2023-04-16T05:11:25.555 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'workload/{ec-rados-default'] 2023-04-16T05:11:25.556 INFO:scrape: 2023-04-16T05:11:25.556 INFO:scrape:Failure: {'smithi142.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 8.43.85.73)", 'rc': 1, 'results': []}, 'smithi073.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 34.221.3.152)", 'rc': 1, 'results': []}} 2023-04-16T05:11:25.556 INFO:scrape:1 jobs: ['7241179'] 2023-04-16T05:11:25.557 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd_api', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8'] 2023-04-16T05:11:25.557 INFO:scrape: 2023-04-16T05:11:25.557 INFO:scrape:Failure: Command failed on smithi117 with status 1: 'sudo yum install -y kernel' 2023-04-16T05:11:25.557 INFO:scrape:1 jobs: ['7241194'] 2023-04-16T05:11:25.558 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'workload/{ec-rados-default'] 2023-04-16T05:11:25.558 INFO:scrape: 2023-04-16T05:11:25.560 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 3 dead: 11 running: 1 waiting: 0 queued: 0 passed: 12 Fail ================================================================= [7241181] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:07:36 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241181/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241181/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00d41b5715344566ab803d6f9e36f78c Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7241194] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241194/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241194/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ebc59cced5449f4a6da33d4ba478411 Command failed on smithi117 with status 1: 'sudo yum install -y kernel' [7241195] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:34 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241195/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f2c3417895a4c34820429d751e5f813 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus Dead ================================================================= [7241173] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:04:24 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241173/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241173/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9207c5810cf4477c9c4f853502930866 {'smithi174.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 38.145.60.21)", 'rc': 1, 'results': []}, 'smithi027.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 38.145.60.21)", 'rc': 1, 'results': []}} [7241176] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:04:23 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241176/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1b416d3ffdc40c7889a8fa86925d927 {'smithi120.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 38.145.60.20)", 'rc': 1, 'results': []}, 'smithi031.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 140.211.169.196)", 'rc': 1, 'results': []}} [7241177] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:04:24 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241177/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241177/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a18b75d892bd4a5089c4050b558d379e {'smithi102.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 34.221.3.152)", 'rc': 1, 'results': []}, 'smithi012.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.198)", 'rc': 1, 'results': []}} [7241179] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:04:18 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241179/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241179/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee15696385814107986d1070d09a838e {'smithi142.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 8.43.85.73)", 'rc': 1, 'results': []}, 'smithi073.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 34.221.3.152)", 'rc': 1, 'results': []}} [7241180] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241180/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241180/ hit max job timeout [7241185] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241185/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241185/ hit max job timeout [7241186] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:04:11 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241186/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee54fed2964c4036ad161e8e948b25e5 {'smithi120.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 38.145.60.21)", 'rc': 1, 'results': []}, 'smithi031.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 38.145.60.21)", 'rc': 1, 'results': []}} [7241190] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:04:08 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241190/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241190/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40f9e99047cf46daa3b1fc66afcd0dde {'smithi129.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.142)", 'rc': 1, 'results': []}, 'smithi099.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 67.219.144.68)", 'rc': 1, 'results': []}} [7241191] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241191/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241191/ hit max job timeout [7241193] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:04:20 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241193/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241193/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b85cde2f47864191892d4632ed402364 {'smithi081.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.198)", 'rc': 1, 'results': []}, 'smithi044.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 8.43.85.73)", 'rc': 1, 'results': []}} [7241202] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:04:16 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241202/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=357870c10e274c51b03d8f304b1373d8 {'smithi129.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.142)", 'rc': 1, 'results': []}, 'smithi202.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 34.221.3.152)", 'rc': 1, 'results': []}} Running ================================================================= [7241200] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241200/ Pass ================================================================= [7241174] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:05:27 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241174/ [7241175] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:33:53 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241175/ [7241178] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:52 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241178/ [7241182] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 03:29:30 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241182/ [7241183] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:11:37 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241183/ [7241184] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:01:16 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241184/ [7241187] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 03:46:46 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241187/ [7241188] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:46:22 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241188/ [7241189] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:36:00 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241189/ [7241192] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:02 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241192/ [7241196] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:41:30 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241196/ [7241198] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 03:57:36 info: http://pulpito.ceph.com/teuthology-2023-04-14_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7241198/