2023-04-20T06:36:43.588 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-20T06:36:43.590 INFO:teuthology.results:subset: '1/13' 2023-04-20T06:36:43.590 INFO:teuthology.results:seed: '9341' 2023-04-20T07:26:34.070 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-20T07:26:34.118 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-20T19:15:27.417 INFO:teuthology.results:Tests finished! gathering results... 2023-04-20T19:15:28.886 INFO:scrape:Found 19 jobs 2023-04-20T19:15:31.463 INFO:scrape:Found 4 distinct failure reasons 2023-04-20T19:15:31.464 INFO:scrape:Failure: Command failed on smithi053 with status 2: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'grep thebody /mnt/foo/myobject'" 2023-04-20T19:15:31.465 INFO:scrape:4 jobs: ['7245987', '7245984', '7245989', '7245977'] 2023-04-20T19:15:31.465 INFO:scrape:suites intersection: [] 2023-04-20T19:15:31.465 INFO:scrape:suites union: ['0-nvme-loop', '1-ranks/1', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '3-final}', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2023-04-20T19:15:31.466 INFO:scrape: 2023-04-20T19:15:31.466 INFO:scrape:Dead: b'2023-04-20T18:51:53.010 DEBUG:teuthology.exit:Finished running handlers' 2023-04-20T19:15:31.466 INFO:scrape:3 jobs: ['7245983', '7245990', '7245978'] 2023-04-20T19:15:31.467 INFO:scrape:suites intersection: ['1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-04-20T19:15:31.467 INFO:scrape:suites union: ['1-bootstrap/16.2.0', '1-bootstrap/16.2.4', '1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-04-20T19:15:31.467 INFO:scrape: 2023-04-20T19:15:31.468 INFO:scrape:Failure: Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ebe00342-df4c-11ed-9b00-001a4aab830c -e sha1=af9762e39fc1916291ffc36ec9c75d97f56722fa -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' 2023-04-20T19:15:31.468 INFO:scrape:2 jobs: ['7245993', '7245982'] 2023-04-20T19:15:31.468 INFO:scrape:suites intersection: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on'] 2023-04-20T19:15:31.469 INFO:scrape:suites union: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-04-20T19:15:31.469 INFO:scrape: 2023-04-20T19:15:31.470 INFO:scrape:Failure: {'smithi132.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}, 'smithi150.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} 2023-04-20T19:15:31.470 INFO:scrape:1 jobs: ['7245994'] 2023-04-20T19:15:31.470 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_orch_cli_mon}'] 2023-04-20T19:15:31.470 INFO:scrape: 2023-04-20T19:15:31.472 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/ failed: 6 dead: 4 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [7245977] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:19:54 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245977/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245977/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a015f2db86744cce8d3bdacf5940aead Command failed on smithi100 with status 2: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'grep thebody /mnt/foo/foobucket/myobject'" [7245982] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 01:26:08 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245982/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245982/ sentry: https://sentry.ceph.com/organizations/ceph/?query=74df9e1c96354774944b0112be5e0658 Command failed on smithi073 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0da1f456-df50-11ed-9b00-001a4aab830c -e sha1=af9762e39fc1916291ffc36ec9c75d97f56722fa -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' [7245984] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245984/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245984/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8a88620894c04b678262c68e0f2518f3 Command failed on smithi081 with status 1: 'sudo yum install -y kernel' [7245987] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:46:54 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245987/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245987/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e03d1043c9124eaeb9b9864bd05ff939 Command failed on smithi053 with status 2: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'grep thebody /mnt/foo/myobject'" [7245989] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:25:20 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245989/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245989/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46d9ee7bbe36466c8fd95a00dcf67dcf Command failed on smithi012 with status 2: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'grep thebody /mnt/foo/foobucket/myobject'" [7245993] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 01:00:24 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245993/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245993/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d9cf970e6974a1abf6c7df0af886a29 Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ebe00342-df4c-11ed-9b00-001a4aab830c -e sha1=af9762e39fc1916291ffc36ec9c75d97f56722fa -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' Dead ================================================================= [7245978] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245978/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245978/ hit max job timeout [7245983] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245983/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245983/ hit max job timeout [7245990] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245990/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245990/ hit max job timeout [7245994] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:38:22 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245994/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245994/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2bcab03ceb414b91a0545fb92d306589 {'smithi132.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24 680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}, 'smithi150.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb- comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} Pass ================================================================= [7245979] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/off orchestrator_cli} time: 01:37:15 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245979/ [7245980] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 01:55:09 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245980/ [7245981] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:54:26 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245981/ [7245985] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:53:40 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245985/ [7245986] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 01:55:54 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245986/ [7245988] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:47:18 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245988/ [7245991] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 01:59:24 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245991/ [7245992] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:56:04 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245992/ [7245995] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:50:42 info: http://pulpito.ceph.com/gabrioux-2023-04-20_06:36:35-orch:cephadm-main-distro-default-smithi/7245995/