2023-03-06T21:04:07.418 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-06T21:04:07.419 INFO:teuthology.results:subset: '1/13' 2023-03-06T21:04:07.419 INFO:teuthology.results:seed: '2714' 2023-03-06T21:04:23.645 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-06T21:04:23.652 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-06T22:16:29.288 INFO:teuthology.results:Tests finished! gathering results... 2023-03-06T22:16:29.699 INFO:scrape:Found 15 jobs 2023-03-06T22:16:30.427 INFO:scrape:Found 5 distinct failure reasons 2023-03-06T22:16:30.428 INFO:scrape:Failure: Command failed on smithi190 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:710f1aa0030e04b9f34e7ffa8424e7dfaaa2752d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c7feb52-bc64-11ed-9aea-001a4aab830c -- bash -c 'ceph nfs cluster create foo --ingress --virtual-ip 10.0.31.190/16'" 2023-03-06T22:16:30.428 INFO:scrape:3 jobs: ['7195402', '7195403', '7195394'] 2023-03-06T22:16:30.429 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2023-03-06T22:16:30.429 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2023-03-06T22:16:30.429 INFO:scrape: 2023-03-06T22:16:30.430 INFO:scrape:Failure: {'smithi140.front.sepia.ceph.com': {'_ansible_no_log': False, 'cache_update_time': 1678137134, 'cache_updated': False, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': True, 'force_apt_get': False, 'install_recommends': None, 'name': ['apt', 'apache2'], 'only_upgrade': False, 'package': ['apt', 'apache2'], 'policy_rc_d': None, 'purge': False, 'state': 'latest', 'update_cache': None, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" --force-yes install \'apache2\'\' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14487 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'rc': 100, 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14487 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14487 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], 'stdout': '', 'stdout_lines': []}} 2023-03-06T22:16:30.430 INFO:scrape:1 jobs: ['7195390'] 2023-03-06T22:16:30.430 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-user', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2023-03-06T22:16:30.430 INFO:scrape: 2023-03-06T22:16:30.431 INFO:scrape:Failure: Command failed on smithi135 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a11d1b5c-bc64-11ed-9aea-001a4aab830c -e sha1=710f1aa0030e04b9f34e7ffa8424e7dfaaa2752d -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' 2023-03-06T22:16:30.431 INFO:scrape:2 jobs: ['7195404', '7195395'] 2023-03-06T22:16:30.431 INFO:scrape:suites intersection: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on'] 2023-03-06T22:16:30.432 INFO:scrape:suites union: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-03-06T22:16:30.432 INFO:scrape: 2023-03-06T22:16:30.432 INFO:scrape:Failure: SSH connection to smithi038 was lost: 'sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64' 2023-03-06T22:16:30.432 INFO:scrape:1 jobs: ['7195399'] 2023-03-06T22:16:30.433 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/basic', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun'] 2023-03-06T22:16:30.433 INFO:scrape: 2023-03-06T22:16:30.433 INFO:scrape:Failure: Command failed on smithi038 with status 1: 'sudo yum install -y kernel' 2023-03-06T22:16:30.434 INFO:scrape:1 jobs: ['7195401'] 2023-03-06T22:16:30.434 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-bucket', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun'] 2023-03-06T22:16:30.434 INFO:scrape: 2023-03-06T22:16:30.435 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/ failed: 6 dead: 2 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7195394] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:14:00 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195394/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195394/ sentry: https://sentry.ceph.com/organizations/ceph/?query=729bd3113944410498491f568228b268 Command failed on smithi016 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:710f1aa0030e04b9f34e7ffa8424e7dfaaa2752d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ccd9e82e-bc64-11ed-9aea-001a4aab830c -- bash -c 'ceph nfs cluster create foo --ingress --virtual-ip 10.0.31.16/16 --port 2999'" [7195395] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:55:20 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195395/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ad6f8a79cf14ec2bb20d53a102b4767 Command failed on smithi039 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b620dac- bc65-11ed-9aea-001a4aab830c -e sha1=710f1aa0030e04b9f34e7ffa8424e7dfaaa2752d -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' [7195401] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195401/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb0b9fc125da444fa8465c903b1f9021 Command failed on smithi038 with status 1: 'sudo yum install -y kernel' [7195402] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:15:26 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195402/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35fe0fee8778458189a74e4019c8f29f Command failed on smithi190 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:710f1aa0030e04b9f34e7ffa8424e7dfaaa2752d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c7feb52-bc64-11ed-9aea-001a4aab830c -- bash -c 'ceph nfs cluster create foo --ingress --virtual-ip 10.0.31.190/16'" [7195403] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:17:19 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195403/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195403/ sentry: https://sentry.ceph.com/organizations/ceph/?query=958e15514ce045daa31a4ce35bbfbe6f Command failed on smithi027 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:710f1aa0030e04b9f34e7ffa8424e7dfaaa2752d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed224162-bc64-11ed-9aea-001a4aab830c -- bash -c 'ceph nfs cluster create foo --ingress --virtual-ip 10.0.31.27/16 --port 2999'" [7195404] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 01:01:21 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195404/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc8e8fca2df44e43a0dc4454bb5b8c09 Command failed on smithi135 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a11d1b5c-bc64-11ed-9aea-001a4aab830c -e sha1=710f1aa0030e04b9f34e7ffa8424e7dfaaa2752d -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\'' Dead ================================================================= [7195390] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:05:12 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195390/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eaae47917efa4a2782cfd00ec6ac5ab0 {'smithi140.front.sepia.ceph.com': {'_ansible_no_log': False, 'cache_update_time': 1678137134, 'cache_updated': False, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': True, 'force_apt_get': False, 'install_recommends': None, 'name': ['apt', 'apache2'], 'only_upgrade': False, 'package': ['apt', 'apache2'], 'policy_rc_d': None, 'purge': False, 'state': 'latest', 'update_cache': None, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" --force-yes install \'apache2\'\' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14487 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'rc': 100, 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14487 (apt- get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock- frontend), is another process using it?\n', 'stderr_lines': ['W: --force- yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14487 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], 'stdout': '', 'stdout_lines': []}} [7195399] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195399/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195399/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdeaf45ab87144d5b39ce621ae692468 SSH connection to smithi038 was lost: 'sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64' Pass ================================================================= [7195391] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:39:59 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195391/ [7195392] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:16:09 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195392/ [7195393] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:16:31 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195393/ [7195396] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:28:19 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195396/ [7195397] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:15:55 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195397/ [7195398] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:13:30 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195398/ [7195400] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:15:40 info: http://pulpito.ceph.com/adking-2023-03-06_21:04:02-orch:cephadm-wip-adk-testing-2023-03-04-1231-distro-default-smithi/7195400/