2022-10-12T05:52:57.950 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-12T05:52:57.952 INFO:teuthology.results:seed: '3619' 2022-10-12T05:56:26.345 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-12T05:56:26.370 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-12T06:39:29.442 INFO:teuthology.results:Tests finished! gathering results... 2022-10-12T06:39:30.033 INFO:scrape:Found 6 jobs 2022-10-12T06:39:32.710 INFO:scrape:Found 2 distinct failure reasons 2022-10-12T06:39:32.711 INFO:scrape:Failure: {'smithi175.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed", 'rc': 1, 'results': []}, 'smithi124.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed", 'rc': 1, 'results': []}} 2022-10-12T06:39:32.711 INFO:scrape:3 jobs: ['7063322', '7063325', '7063324'] 2022-10-12T06:39:32.712 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/metrics}'] 2022-10-12T06:39:32.713 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/metrics}'] 2022-10-12T06:39:32.713 INFO:scrape: 2022-10-12T06:39:32.714 INFO:scrape:Failure: "1665555560.0768254 mon.a (mon.0) 2098 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-10-12T06:39:32.714 INFO:scrape:2 jobs: ['7063326', '7063323'] 2022-10-12T06:39:32.715 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/metrics}'] 2022-10-12T06:39:32.715 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/metrics}'] 2022-10-12T06:39:32.716 INFO:scrape: 2022-10-12T06:39:32.717 INFO:teuthology.results:Sending results to neesingh@redhat.com: Test Run: neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/ failed: 3 dead: 2 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7063323] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:28:21 info: http://pulpito.ceph.com/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063323/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063323/ "1665555874.5261245 mon.a (mon.0) 2722 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7063324] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:07:20 info: http://pulpito.ceph.com/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063324/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063324/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a82da71adfa47c28f56fca0ce2b3d01 {'smithi106.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed", 'rc': 1, 'results': []}, 'smithi049.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed", 'rc': 1, 'results': []}} [7063326] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:28:48 info: http://pulpito.ceph.com/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063326/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063326/ "1665555560.0768254 mon.a (mon.0) 2098 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log Dead ================================================================= [7063322] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:04:06 info: http://pulpito.ceph.com/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063322/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063322/ sentry: https://sentry.ceph.com/organizations/ceph/?query=830d3a417c21458098edd5413194d6fe {'smithi175.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed", 'rc': 1, 'results': []}, 'smithi124.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed", 'rc': 1, 'results': []}} [7063325] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:04:07 info: http://pulpito.ceph.com/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063325/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063325/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f093d9bcb8540f38ca5e9d7a0157da9 {'smithi107.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed", 'rc': 1, 'results': []}, 'smithi017.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed", 'rc': 1, 'results': []}} Pass ================================================================= [7063327] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:32:02 info: http://pulpito.ceph.com/neesingh-2022-10-12_05:50:40-fs-wip-tracker-54557-distro-default-smithi/7063327/