2024-04-13T09:56:02.176 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-13T09:56:02.176 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-08_22:08:01-orch-reef-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-08_22:08:01-orch-reef-distro-default-smithi/7646569 --verbose --description orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} -- /home/teuthworker/archive/teuthology-2024-04-08_22:08:01-orch-reef-distro-default-smithi/7646569/orig.config.yaml 2024-04-13T09:56:02.201 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-13T09:56:02.279 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-08_22:08:01-orch-reef-distro-default-smithi/7646569 branch: reef description: orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7646569' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-08_22:08:01-orch-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: reef ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5108bc67b831101bc32a1061150595384640c56c ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5108bc67b831101bc32a1061150595384640c56c selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 seed: 4929 sha1: 5108bc67b831101bc32a1061150595384640c56c sleep_before_teardown: 0 subset: 11/64 suite: orch suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxm6ND1iicJsSEPBhlCyhJE/ZorHpXyrtHCOGbzcfRwZQjDgNqS38NQ3jLL9AaHhiCK50JmpVBLAza5Y0hUM5c= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - cephadm: roleless: true single_host_defaults: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-08_22:08:01 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-13T09:56:02.280 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa; will attempt to use it 2024-04-13T09:56:02.280 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks 2024-04-13T09:56:02.280 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-13T09:56:02.282 INFO:teuthology.task.internal:Checking packages... 2024-04-13T09:56:02.304 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5108bc67b831101bc32a1061150595384640c56c' 2024-04-13T09:56:02.304 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-13T09:56:02.304 INFO:teuthology.packaging:ref: None 2024-04-13T09:56:02.304 INFO:teuthology.packaging:tag: None 2024-04-13T09:56:02.304 INFO:teuthology.packaging:branch: reef 2024-04-13T09:56:02.304 INFO:teuthology.packaging:sha1: 5108bc67b831101bc32a1061150595384640c56c 2024-04-13T09:56:02.305 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef 2024-04-13T09:56:02.527 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1217.g7d36322c 2024-04-13T09:56:02.528 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-13T09:56:02.535 INFO:teuthology.task.internal:no buildpackages task found 2024-04-13T09:56:02.535 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-13T09:56:02.545 INFO:teuthology.task.internal:Saving configuration 2024-04-13T09:56:02.557 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-13T09:56:02.563 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T09:56:02.588 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_22:08:01-orch-reef-distro-default-smithi/7646569', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 09:50:58.775206', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxm6ND1iicJsSEPBhlCyhJE/ZorHpXyrtHCOGbzcfRwZQjDgNqS38NQ3jLL9AaHhiCK50JmpVBLAza5Y0hUM5c='} 2024-04-13T09:56:02.588 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-13T09:56:02.594 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-13T09:56:02.594 INFO:teuthology.run_tasks:Running task console_log... 2024-04-13T09:56:02.645 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3e1bf75280>, signals=[15]) 2024-04-13T09:56:02.645 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-13T09:56:02.651 INFO:teuthology.task.internal:Opening connections... 2024-04-13T09:56:02.651 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-04-13T09:56:02.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T09:56:02.729 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-13T09:56:02.735 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-04-13T09:56:02.800 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-04-13T09:56:02.801 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-04-13T09:56:02.861 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-04-13T09:56:02.861 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="8" 2024-04-13T09:56:02.861 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-04-13T09:56:02.861 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-04-13T09:56:02.861 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="8" 2024-04-13T09:56:02.862 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el8" 2024-04-13T09:56:02.862 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-13T09:56:02.862 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-04-13T09:56:02.862 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-13T09:56:02.862 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-04-13T09:56:02.862 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-13T09:56:02.862 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-13T09:56:02.862 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-13T09:56:02.863 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-04-13T09:56:02.884 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-13T09:56:02.892 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-13T09:56:02.963 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-13T09:56:02.963 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-04-13T09:56:02.983 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-13T09:56:02.989 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-13T09:56:02.990 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-04-13T09:56:03.054 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-13T09:56:03.086 INFO:teuthology.run_tasks:Running task kernel... 2024-04-13T09:56:03.101 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-13T09:56:03.101 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-13T09:56:03.101 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-13T09:56:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T09:56:03.121 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-13T09:56:03.180 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-489.el8.x86_64 2024-04-13T09:56:03.181 INFO:teuthology.task.kernel:Running kernel on smithi134: 4.18.0-489.el8.x86_64 2024-04-13T09:56:03.181 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-04-13T09:56:06.793 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - AppStream 25 MB/s | 28 MB 00:01 2024-04-13T09:56:12.676 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-04-13T09:56:14.701 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2024-04-13T09:56:15.352 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 7.7 kB 00:00 2024-04-13T09:56:15.704 INFO:teuthology.orchestra.run.smithi134.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2024-04-13T09:56:16.421 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2024-04-13T09:56:20.171 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 449 kB/s | 24 kB 00:00 2024-04-13T09:56:23.159 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-13T09:56:23.160 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-13T09:56:23.275 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-13T09:56:23.275 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-13T09:56:23.275 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-13T09:56:23.275 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-13T09:56:23.275 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout:Install 3 Packages 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 90 M 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 96 M 2024-04-13T09:56:23.276 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-13T09:56:25.214 INFO:teuthology.orchestra.run.smithi134.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 6.0 MB/s | 10 MB 00:01 2024-04-13T09:56:28.597 INFO:teuthology.orchestra.run.smithi134.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 7.1 MB/s | 36 MB 00:05 2024-04-13T09:56:29.181 INFO:teuthology.orchestra.run.smithi134.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 7.7 MB/s | 44 MB 00:05 2024-04-13T09:56:29.181 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-13T09:56:29.181 INFO:teuthology.orchestra.run.smithi134.stdout:Total 15 MB/s | 90 MB 00:05 2024-04-13T09:56:29.866 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-13T09:56:30.290 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-13T09:56:30.291 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-13T09:56:31.550 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-13T09:56:31.572 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-13T09:56:34.335 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-13T09:56:34.900 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-13T09:56:38.458 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-13T09:56:38.833 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-13T09:56:42.275 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-13T09:56:42.557 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-13T09:57:04.431 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-13T09:57:04.456 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-13T09:57:04.798 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-13T09:57:04.798 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-13T09:57:04.798 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-13T09:57:05.286 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-13T09:57:05.287 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-13T09:57:05.287 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-13T09:57:05.287 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-13T09:57:05.287 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-13T09:57:05.287 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-13T09:57:05.287 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-13T09:57:05.471 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-04-13T09:57:05.688 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-13T09:57:05.689 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-13T09:57:05.689 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-13T09:57:05.689 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-13T09:57:05.689 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-13T09:57:05.689 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-13T09:57:05.689 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-04-13T09:57:05.690 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-04-13T09:57:05.901 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-13T09:57:05.901 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-13T09:57:05.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-04-13T09:57:05.935 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-04-13T09:57:05.935 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-04-13T09:57:05.935 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.02 2024-04-13T09:57:05.935 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 129.el8 2024-04-13T09:57:05.935 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-04-13T09:57:05.935 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Group : System Environment/Base 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 9265586 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Relocations : (not relocatable) 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : CentOS BuildSystem 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-13T09:57:05.936 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-04-13T09:57:05.937 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-04-13T09:57:05.937 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-13T09:57:05.937 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-13T09:57:05.937 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-13T09:57:05.937 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-13T09:57:05.937 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-04-13T09:57:05.937 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-13T09:57:05.937 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-04-13T09:57:05.938 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-13T09:57:05.938 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 4.18.0-552.el8.x86_64 2024-04-13T09:57:05.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-13T09:57:07.556 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-04-13T09:57:09.407 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-04-13T09:57:09.409 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-04-13T09:57:09.439 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.cGPuPdD9rt 2024-04-13T09:57:09.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cGPuPdD9rt 2024-04-13T09:57:09.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.cGPuPdD9rt 2024-04-13T09:57:09.668 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.cGPuPdD9rt is 6KB 2024-04-13T09:57:09.719 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.cGPuPdD9rt 2024-04-13T09:57:09.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-04-13T09:57:09.821 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-13T09:57:09.821 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-13T09:57:09.822 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-13T09:57:09.822 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-13T09:57:09.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-13T09:57:09.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-04-13T09:57:39.968 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-13T09:57:39.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-13T09:57:39.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T09:57:58.431 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-04-13T09:58:07.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-13T09:58:07.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T09:58:10.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-04-13T09:58:22.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-13T09:58:22.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T09:58:25.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-04-13T09:58:40.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-13T09:58:40.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T09:58:43.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-04-13T09:59:01.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-13T09:59:01.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T09:59:02.067 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-04-13T09:59:02.387 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-13T09:59:02.387 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-552.el8.x86_64"... 2024-04-13T09:59:02.387 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-13T09:59:02.405 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-552.el8.x86_64 2024-04-13T09:59:02.405 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-13T09:59:02.406 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-13T09:59:02.406 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-13T09:59:03.407 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-13T09:59:03.407 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-13T09:59:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-04-13T09:59:03.488 DEBUG:teuthology.parallel:result is None 2024-04-13T09:59:03.489 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-13T09:59:03.496 INFO:teuthology.task.internal:Creating test directory... 2024-04-13T09:59:03.496 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-13T09:59:03.516 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-13T09:59:03.540 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-13T09:59:03.550 INFO:teuthology.task.internal:Creating archive directory... 2024-04-13T09:59:03.550 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-13T09:59:03.591 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-13T09:59:03.598 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-13T09:59:03.598 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-13T09:59:03.666 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T09:59:03.676 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T09:59:03.677 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-13T09:59:03.685 INFO:teuthology.task.internal:Configuring sudo... 2024-04-13T09:59:03.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-13T09:59:03.747 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-13T09:59:03.757 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-13T09:59:03.757 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-13T09:59:03.807 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T09:59:03.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T09:59:03.944 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T09:59:04.028 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T09:59:04.074 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-13T09:59:04.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-13T09:59:04.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-04-13T09:59:04.231 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T09:59:04.581 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-13T09:59:04.589 INFO:teuthology.task.internal:Starting timer... 2024-04-13T09:59:04.589 INFO:teuthology.run_tasks:Running task pcp... 2024-04-13T09:59:04.599 INFO:teuthology.run_tasks:Running task selinux... 2024-04-13T09:59:04.607 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-13T09:59:04.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-04-13T09:59:04.718 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-04-13T09:59:04.720 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-13T09:59:04.720 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-04-13T09:59:04.746 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-04-13T09:59:04.747 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-04-13T09:59:04.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T09:59:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T09:59:04.814 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-13T09:59:04.815 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-04-13T09:59:04.890 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-13T09:59:04.901 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-13T09:59:04.919 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-13T09:59:04.923 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-13T10:03:31.627 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/teuthology-2024-04-08_22:08:01-orch-reef-distro-default-smithi/7646569/ansible_failures.yaml 2024-04-13T10:03:31.658 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-13T10:03:32.976 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1ee70fbe62aa4dc8b481fa12028a28a0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-13T10:03:32.979 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-13T10:03:32.987 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-13T10:03:32.987 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-13T10:03:32.994 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-13T10:03:33.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-13T10:03:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T10:03:33.091 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-13T10:03:33.099 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-13T10:03:33.108 INFO:teuthology.task.internal:Duration was 268.519386 seconds 2024-04-13T10:03:33.109 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-13T10:03:33.118 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-13T10:03:33.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-13T10:03:33.177 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T10:03:33.423 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-13T10:03:33.423 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-04-13T10:03:33.424 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-13T10:03:33.445 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-13T10:03:33.446 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T10:03:33.525 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-13T10:03:33.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T10:03:33.639 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-13T10:03:33.652 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-13T10:03:33.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-13T10:03:33.679 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-13T10:03:33.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-13T10:03:33.746 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-04-13T10:03:33.770 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-13T10:03:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T10:03:33.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-13T10:03:33.836 INFO:teuthology.task.internal:Transferring archived files... 2024-04-13T10:03:33.838 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-08_22:08:01-orch-reef-distro-default-smithi/7646569/remote/smithi134 2024-04-13T10:03:33.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-13T10:03:33.912 INFO:teuthology.task.internal:Removing archive directory... 2024-04-13T10:03:33.912 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-13T10:03:33.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-13T10:03:33.979 INFO:teuthology.task.internal:Not uploading archives. 2024-04-13T10:03:33.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-13T10:03:33.988 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-13T10:03:33.988 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-13T10:03:34.026 INFO:teuthology.orchestra.run.smithi134.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 10:03 /home/ubuntu/cephtest 2024-04-13T10:03:34.027 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-13T10:03:34.041 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-13T10:03:34.076 INFO:teuthology.nuke:Checking targets against current locks 2024-04-13T10:03:34.106 DEBUG:teuthology.nuke:shortname: smithi134 2024-04-13T10:03:34.106 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T10:03:34.130 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_22:08:01-orch-reef-distro-default-smithi/7646569', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 09:50:58.775206', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxm6ND1iicJsSEPBhlCyhJE/ZorHpXyrtHCOGbzcfRwZQjDgNqS38NQ3jLL9AaHhiCK50JmpVBLAza5Y0hUM5c='} 2024-04-13T10:03:34.188 INFO:teuthology.orchestra.console.smithi134:Power off 2024-04-13T10:03:34.188 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-13T10:03:34.213 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-04-13T10:03:34.213 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T10:03:34.242 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-04-13T10:03:38.243 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T10:03:38.354 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-04-13T10:03:42.358 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T10:03:42.468 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-04-13T10:03:46.473 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T10:03:46.583 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-04-13T10:03:46.684 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-04-13T10:03:46.791 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} duration: 268.51938557624817 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1ee70fbe62aa4dc8b481fa12028a28a0 status: dead success: false 2024-04-13T10:03:46.791 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-13T10:03:46.887 INFO:teuthology.run:DEAD