2023-12-18T13:30:01.935 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-18T13:30:01.935 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/adking-2023-12-18_13:16:52-orch:cephadm-wip-adk-testing-2023-12-17-2227-distro-default-smithi/7495508 --name adking-2023-12-18_13:16:52-orch:cephadm-wip-adk-testing-2023-12-17-2227-distro-default-smithi --description orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli} --verbose --owner scheduled_adking@teuthology -- /home/teuthworker/archive/adking-2023-12-18_13:16:52-orch:cephadm-wip-adk-testing-2023-12-17-2227-distro-default-smithi/7495508/orig.config.yaml 2023-12-18T13:30:01.965 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-18T13:30:02.052 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2023-12-18_13:16:52-orch:cephadm-wip-adk-testing-2023-12-17-2227-distro-default-smithi/7495508 branch: wip-adk-testing-2023-12-17-2227 description: orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli} email: adking@redhat.com first_in_suite: false job_id: '7495508' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: adking-2023-12-18_13:16:52-orch:cephadm-wip-adk-testing-2023-12-17-2227-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-adk-testing-2023-12-17-2227 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 861525e252e70f0d455bae7d2dc4d635f8b415f1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 861525e252e70f0d455bae7d2dc4d635f8b415f1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-adk-testing-2023-12-17-2227 sha1: 861525e252e70f0d455bae7d2dc4d635f8b415f1 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mgr.x - mon.a - mon.c - mds.a - mds.c - osd.0 - client.0 - - mgr.y - mgr.z - mon.b - mds.b - osd.1 - osd.2 - osd.3 - client.1 seed: 6505 sha1: 861525e252e70f0d455bae7d2dc4d635f8b415f1 sleep_before_teardown: 0 subset: 1/14 suite: orch:cephadm suite_branch: wip-adk-testing-2023-12-17-2227 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_861525e252e70f0d455bae7d2dc4d635f8b415f1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 861525e252e70f0d455bae7d2dc4d635f8b415f1 targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBMmpwZL1whmHjGnfN/8EASp5iG/xx1kO4Z/yO2aXH0rZg/ZzAHBeVsImlST8T/CzjcrBhKuFm1YvOYaN2vVt9o= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEDsUFh3DIn12vfL+ji8bN7LIUx5tbWnEQSDChwMUaFyKeQDoYP/r+oF8BwWS9Jpqsh88/ZR0qf2aMVh+iw6KIk= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - ceph: log-ignorelist: - overall HEALTH_ - \(MGR_DOWN\) - \(DEVICE_IDENT_ON\) - \(DEVICE_FAULT_ON\) - \(PG_ - replacing it with standby - No standby daemons available - \(POOL_APP_NOT_ENABLED\) wait-for-scrub: false - cephfs_test_runner: modules: - tasks.mgr.test_orchestrator_cli teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-18_13:16:52 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-18T13:30:02.052 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_861525e252e70f0d455bae7d2dc4d635f8b415f1/qa; will attempt to use it 2023-12-18T13:30:02.053 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_861525e252e70f0d455bae7d2dc4d635f8b415f1/qa/tasks 2023-12-18T13:30:02.053 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-18T13:30:02.054 INFO:teuthology.task.internal:Checking packages... 2023-12-18T13:30:02.092 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '861525e252e70f0d455bae7d2dc4d635f8b415f1' 2023-12-18T13:30:02.092 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-18T13:30:02.092 INFO:teuthology.packaging:ref: None 2023-12-18T13:30:02.092 INFO:teuthology.packaging:tag: None 2023-12-18T13:30:02.092 INFO:teuthology.packaging:branch: wip-adk-testing-2023-12-17-2227 2023-12-18T13:30:02.093 INFO:teuthology.packaging:sha1: 861525e252e70f0d455bae7d2dc4d635f8b415f1 2023-12-18T13:30:02.093 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-adk-testing-2023-12-17-2227 2023-12-18T13:30:02.243 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-303.g861525e2 2023-12-18T13:30:02.245 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-18T13:30:02.251 INFO:teuthology.task.internal:no buildpackages task found 2023-12-18T13:30:02.251 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-18T13:30:02.267 INFO:teuthology.task.internal:Saving configuration 2023-12-18T13:30:02.278 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-18T13:30:02.288 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-18T13:30:02.333 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-12-18_13:16:52-orch:cephadm-wip-adk-testing-2023-12-17-2227-distro-default-smithi/7495508', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-18 13:24:58.597669', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBMmpwZL1whmHjGnfN/8EASp5iG/xx1kO4Z/yO2aXH0rZg/ZzAHBeVsImlST8T/CzjcrBhKuFm1YvOYaN2vVt9o='} 2023-12-18T13:30:02.365 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-12-18_13:16:52-orch:cephadm-wip-adk-testing-2023-12-17-2227-distro-default-smithi/7495508', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-18 13:24:58.596471', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEDsUFh3DIn12vfL+ji8bN7LIUx5tbWnEQSDChwMUaFyKeQDoYP/r+oF8BwWS9Jpqsh88/ZR0qf2aMVh+iw6KIk='} 2023-12-18T13:30:02.365 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-18T13:30:02.374 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-12-18T13:30:02.374 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-12-18T13:30:02.374 INFO:teuthology.run_tasks:Running task console_log... 2023-12-18T13:30:02.528 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe3b6b98b80>, signals=[15]) 2023-12-18T13:30:02.529 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-18T13:30:02.542 INFO:teuthology.task.internal:Opening connections... 2023-12-18T13:30:02.542 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2023-12-18T13:30:02.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-18T13:30:02.614 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2023-12-18T13:30:02.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-18T13:30:02.619 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 387, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 381, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.154 2023-12-18T13:30:02.755 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=142f213277b84166bb0a1138fdb6b6a5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 387, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 381, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.154 2023-12-18T13:30:02.757 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-18T13:30:02.782 INFO:teuthology.nuke:Checking targets against current locks 2023-12-18T13:30:02.822 DEBUG:teuthology.nuke:shortname: smithi055 2023-12-18T13:30:02.823 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-18T13:30:02.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-12-18_13:16:52-orch:cephadm-wip-adk-testing-2023-12-17-2227-distro-default-smithi/7495508', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-18 13:24:58.597669', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBMmpwZL1whmHjGnfN/8EASp5iG/xx1kO4Z/yO2aXH0rZg/ZzAHBeVsImlST8T/CzjcrBhKuFm1YvOYaN2vVt9o='} 2023-12-18T13:30:02.850 DEBUG:teuthology.nuke:shortname: smithi154 2023-12-18T13:30:02.851 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-18T13:30:02.876 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-12-18_13:16:52-orch:cephadm-wip-adk-testing-2023-12-17-2227-distro-default-smithi/7495508', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-18 13:24:58.596471', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEDsUFh3DIn12vfL+ji8bN7LIUx5tbWnEQSDChwMUaFyKeQDoYP/r+oF8BwWS9Jpqsh88/ZR0qf2aMVh+iw6KIk='} 2023-12-18T13:30:02.944 INFO:teuthology.orchestra.console.smithi055:Power off 2023-12-18T13:30:02.945 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-18T13:30:03.010 INFO:teuthology.orchestra.console.smithi154:Power off 2023-12-18T13:30:03.010 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-18T13:30:03.029 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2023-12-18T13:30:03.029 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-18T13:30:03.056 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2023-12-18T13:30:03.057 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-18T13:30:03.079 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-12-18T13:30:03.103 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2023-12-18T13:30:07.081 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-18T13:30:07.104 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-18T13:30:07.201 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-12-18T13:30:07.235 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2023-12-18T13:30:07.337 INFO:teuthology.orchestra.console.smithi154:Power off completed 2023-12-18T13:30:11.206 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-18T13:30:11.327 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-12-18T13:30:15.332 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-18T13:30:15.473 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2023-12-18T13:30:15.573 INFO:teuthology.orchestra.console.smithi055:Power off completed 2023-12-18T13:30:15.681 INFO:teuthology.run:Summary data: description: orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli} failure_reason: '[Errno None] Unable to connect to port 22 on 172.21.15.154' owner: scheduled_adking@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=142f213277b84166bb0a1138fdb6b6a5 status: fail success: false 2023-12-18T13:30:15.681 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-18T13:30:15.769 INFO:teuthology.run:FAIL