2023-04-15T03:26:47.389 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-15T03:26:47.391 INFO:teuthology.results:subset: '111/12000' 2023-04-15T03:26:47.391 INFO:teuthology.results:seed: '1940' 2023-04-15T03:46:38.240 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-15T03:46:38.270 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-15T07:43:56.703 INFO:teuthology.results:Tests finished! gathering results... 2023-04-15T07:43:57.465 INFO:scrape:Found 9 jobs 2023-04-15T07:43:57.698 INFO:scrape:Found 2 distinct failure reasons 2023-04-15T07:43:57.699 INFO:scrape:Failure: {'smithi139.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.198)", 'rc': 1, 'results': []}, 'smithi165.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 34.221.3.152)", 'rc': 1, 'results': []}} 2023-04-15T07:43:57.699 INFO:scrape:3 jobs: ['7241868', '7241874', '7241862'] 2023-04-15T07:43:57.699 INFO:scrape:suites intersection: [] 2023-04-15T07:43:57.700 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'workload/{ec-rados-default'] 2023-04-15T07:43:57.700 INFO:scrape: 2023-04-15T07:43:57.700 INFO:scrape:Failure: {'smithi059.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 38.145.60.21)", 'rc': 1, 'results': []}, 'smithi037.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.198)", 'rc': 1, 'results': []}} 2023-04-15T07:43:57.701 INFO:scrape:1 jobs: ['7241872'] 2023-04-15T07:43:57.701 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd-import-export', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0'] 2023-04-15T07:43:57.701 INFO:scrape: 2023-04-15T07:43:57.702 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/ failed: 0 dead: 4 running: 0 waiting: 0 queued: 0 passed: 5 Dead ================================================================= [7241862] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:04:36 info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241862/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241862/ sentry: https://sentry.ceph.com/organizations/ceph/?query=372a73ed66d34c7faafaa3db1d027c7c {'smithi129.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.142)", 'rc': 1, 'results': []}, 'smithi099.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 140.211.169.196)", 'rc': 1, 'results': []}} [7241868] upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:04:36 info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241868/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=073e8dd83da24cbcb065e60cb53973f1 {'smithi139.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.198)", 'rc': 1, 'results': []}, 'smithi165.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 34.221.3.152)", 'rc': 1, 'results': []}} [7241872] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:04:49 info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241872/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241872/ sentry: https://sentry.ceph.com/organizations/ceph/?query=adba30bec2c848b78826b9b3d8f4ea7a {'smithi059.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 38.145.60.21)", 'rc': 1, 'results': []}, 'smithi037.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 152.19.134.198)", 'rc': 1, 'results': []}} [7241874] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:04:39 info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241874/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241874/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9bd9811f252454fb2cc4d929f550235 {'smithi110.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 209.132.190.2)", 'rc': 1, 'results': []}, 'smithi040.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org /metalink?repo=epel-8.6&arch=x86_64&infra=$infra&content=$contentdir (IP: 34.221.3.152)", 'rc': 1, 'results': []}} Pass ================================================================= [7241864] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:32:08 info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241864/ [7241866] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 03:30:19 info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241866/ [7241870] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 03:34:19 info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241870/ [7241876] upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 02:47:30 info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241876/ [7241878] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:15:43 info: http://pulpito.ceph.com/yuriw-2023-04-14_22:04:53-upgrade:pacific-x-reef-distro-default-smithi/7241878/