2023-12-12T00:32:09.785 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-12T00:32:09.786 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487624 --verbose --owner scheduled_yuriw@teuthology --description rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} -- /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487624/orig.config.yaml 2023-12-12T00:32:09.816 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-12T00:32:09.897 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487624 branch: wip-yuri8-testing-2023-12-11-1101 description: rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} email: yweinste@redhat.com first_in_suite: false job_id: '7487624' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-12-11-1101 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri8-testing-2023-12-11-1101 sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - osd.0 - osd.1 - osd.2 - mon.a - mgr.a - client.0 seed: 3273 sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2023-12-11-1101 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_21b8309530d48c7266e1db5c3ccbe10963f99737/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 targets: smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGDjZsN84llzgWCxN68NZGt8z00PwrXciT+xNGbTgUbvnID9Uaucgi9ggS34RLDrvkvewKEtCKaCaG4RTmF19jY= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: null - cephadm.shell: host.a: - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - workunit: clients: client.0: - cephadm/test_iscsi_pids_limit.sh - cephadm/test_iscsi_etc_hosts.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-11_23:27:14 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-12T00:32:09.897 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_21b8309530d48c7266e1db5c3ccbe10963f99737/qa; will attempt to use it 2023-12-12T00:32:09.898 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_21b8309530d48c7266e1db5c3ccbe10963f99737/qa/tasks 2023-12-12T00:32:09.898 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-12T00:32:09.900 INFO:teuthology.task.internal:Checking packages... 2023-12-12T00:32:09.927 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b8309530d48c7266e1db5c3ccbe10963f99737' 2023-12-12T00:32:09.928 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-12T00:32:09.928 INFO:teuthology.packaging:ref: None 2023-12-12T00:32:09.928 INFO:teuthology.packaging:tag: None 2023-12-12T00:32:09.928 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-12-11-1101 2023-12-12T00:32:09.928 INFO:teuthology.packaging:sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 2023-12-12T00:32:09.928 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri8-testing-2023-12-11-1101 2023-12-12T00:32:10.071 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-130.g21b83095 2023-12-12T00:32:10.073 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-12T00:32:10.081 INFO:teuthology.task.internal:no buildpackages task found 2023-12-12T00:32:10.082 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-12T00:32:10.110 INFO:teuthology.task.internal:Saving configuration 2023-12-12T00:32:10.121 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-12T00:32:10.161 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-12T00:32:10.185 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487624', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 00:26:54.057352', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGDjZsN84llzgWCxN68NZGt8z00PwrXciT+xNGbTgUbvnID9Uaucgi9ggS34RLDrvkvewKEtCKaCaG4RTmF19jY='} 2023-12-12T00:32:10.185 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-12T00:32:10.210 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-12-12T00:32:10.210 INFO:teuthology.run_tasks:Running task console_log... 2023-12-12T00:32:10.306 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2cfca34e50>, signals=[15]) 2023-12-12T00:32:10.307 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-12T00:32:10.319 INFO:teuthology.task.internal:Opening connections... 2023-12-12T00:32:10.319 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2023-12-12T00:32:10.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T00:32:10.391 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-12T00:32:10.405 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2023-12-12T00:32:10.426 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2023-12-12T00:32:10.426 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2023-12-12T00:32:10.487 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-12T00:32:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-12T00:32:10.489 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2023-12-12T00:32:10.508 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-12T00:32:10.523 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-12T00:32:10.620 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-12T00:32:10.620 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2023-12-12T00:32:10.641 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-12T00:32:10.666 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-12T00:32:10.666 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2023-12-12T00:32:10.714 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-12T00:32:10.752 INFO:teuthology.run_tasks:Running task kernel... 2023-12-12T00:32:10.807 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-12T00:32:10.807 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-12T00:32:10.807 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-12T00:32:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T00:32:10.829 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2023-12-12T00:32:10.890 INFO:teuthology.orchestra.run.smithi162.stdout:4.18.0-489.el8.x86_64 2023-12-12T00:32:10.890 INFO:teuthology.task.kernel:Running kernel on smithi162: 4.18.0-489.el8.x86_64 2023-12-12T00:32:10.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2023-12-12T00:32:14.251 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - AppStream 38 MB/s | 35 MB 00:00 2023-12-12T00:32:21.651 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - BaseOS 52 MB/s | 56 MB 00:01 2023-12-12T00:32:29.825 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras 82 kB/s | 18 kB 00:00 2023-12-12T00:32:30.267 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 6.9 kB 00:00 2023-12-12T00:32:30.552 INFO:teuthology.orchestra.run.smithi162.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-12-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-12-12T00:32:35.494 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 443 kB/s | 24 kB 00:00 2023-12-12T00:32:39.118 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-12T00:32:39.118 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-12T00:32:39.413 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 4.18.0-527.el8 baseos 10 M 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 4.18.0-527.el8 baseos 43 M 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 4.18.0-527.el8 baseos 36 M 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-12T00:32:39.414 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2023-12-12T00:32:39.415 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2023-12-12T00:32:39.415 INFO:teuthology.orchestra.run.smithi162.stdout:Install 3 Packages 2023-12-12T00:32:39.415 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-12T00:32:39.415 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 90 M 2023-12-12T00:32:39.415 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 96 M 2023-12-12T00:32:39.415 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2023-12-12T00:32:40.195 INFO:teuthology.orchestra.run.smithi162.stdout:(1/3): kernel-4.18.0-527.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2023-12-12T00:32:41.323 INFO:teuthology.orchestra.run.smithi162.stdout:(2/3): kernel-modules-4.18.0-527.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2023-12-12T00:32:41.779 INFO:teuthology.orchestra.run.smithi162.stdout:(3/3): kernel-core-4.18.0-527.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2023-12-12T00:32:41.779 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2023-12-12T00:32:41.779 INFO:teuthology.orchestra.run.smithi162.stdout:Total 38 MB/s | 90 MB 00:02 2023-12-12T00:32:42.463 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2023-12-12T00:32:42.888 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2023-12-12T00:32:42.888 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2023-12-12T00:32:44.143 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2023-12-12T00:32:44.154 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2023-12-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2023-12-12T00:32:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-4.18.0-527.el8.x86_64 1/3 2023-12-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-4.18.0-527.el8.x86_64 1/3 2023-12-12T00:32:51.525 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-4.18.0-527.el8.x86_64 2/3 2023-12-12T00:32:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-4.18.0-527.el8.x86_64 2/3 2023-12-12T00:32:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-4.18.0-527.el8.x86_64 3/3 2023-12-12T00:33:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-4.18.0-527.el8.x86_64 3/3 2023-12-12T00:33:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-4.18.0-527.el8.x86_64 3/3 2023-12-12T00:33:17.464 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-4.18.0-527.el8.x86_64 3/3 2023-12-12T00:33:17.464 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-4.18.0-527.el8.x86_64 1/3 2023-12-12T00:33:17.464 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-4.18.0-527.el8.x86_64 2/3 2023-12-12T00:33:18.304 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-4.18.0-527.el8.x86_64 3/3 2023-12-12T00:33:18.304 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-12T00:33:18.304 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2023-12-12T00:33:18.304 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-4.18.0-527.el8.x86_64 kernel-core-4.18.0-527.el8.x86_64 2023-12-12T00:33:18.304 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-4.18.0-527.el8.x86_64 2023-12-12T00:33:18.304 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-12T00:33:18.304 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2023-12-12T00:33:18.480 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2023-12-12T00:33:18.696 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-4.18.0-527.el8.x86_64 2023-12-12T00:33:18.696 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-527.el8.x86_64 2023-12-12T00:33:18.697 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-527.el8.x86_64 2023-12-12T00:33:18.697 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-12T00:33:18.697 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-12T00:33:18.697 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-12T00:33:18.697 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2023-12-12T00:33:18.697 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2023-12-12T00:33:18.906 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-4.18.0-527.el8.x86_64 2023-12-12T00:33:18.907 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-527.el8.x86_64 2023-12-12T00:33:18.907 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2023-12-12T00:33:18.942 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2023-12-12T00:33:18.942 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2023-12-12T00:33:18.942 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.02 2023-12-12T00:33:18.942 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 129.el8 2023-12-12T00:33:18.942 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2023-12-12T00:33:18.943 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-12T00:33:18.943 INFO:teuthology.orchestra.run.smithi162.stdout:Group : System Environment/Base 2023-12-12T00:33:18.943 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 9265586 2023-12-12T00:33:18.943 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2023-12-12T00:33:18.943 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-12T00:33:18.943 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-12T00:33:18.943 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-12T00:33:18.943 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-12T00:33:18.943 INFO:teuthology.orchestra.run.smithi162.stdout:Relocations : (not relocatable) 2023-12-12T00:33:18.944 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : CentOS BuildSystem 2023-12-12T00:33:18.944 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2023-12-12T00:33:18.944 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-12T00:33:18.944 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2023-12-12T00:33:18.944 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2023-12-12T00:33:18.944 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-12T00:33:18.944 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-12T00:33:18.944 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-12T00:33:18.945 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-12T00:33:18.945 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2023-12-12T00:33:18.945 INFO:teuthology.orchestra.run.smithi162.stdout: 2023-12-12T00:33:18.945 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2023-12-12T00:33:18.946 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-12T00:33:18.946 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 4.18.0-527.el8.x86_64 2023-12-12T00:33:18.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-12T00:33:20.628 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2023-12-12T00:33:22.803 INFO:teuthology.orchestra.run.smithi162.stderr:done 2023-12-12T00:33:22.804 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2023-12-12T00:33:22.840 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.BakNvDpdQH 2023-12-12T00:33:22.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BakNvDpdQH 2023-12-12T00:33:22.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.BakNvDpdQH 2023-12-12T00:33:23.064 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.BakNvDpdQH is 6KB 2023-12-12T00:33:23.114 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.BakNvDpdQH 2023-12-12T00:33:23.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2023-12-12T00:33:23.198 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-12T00:33:23.198 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-12T00:33:23.198 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-12T00:33:23.198 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-527.el8.x86_64.conf 2023-12-12T00:33:23.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-527.el8.x86_64 2023-12-12T00:33:23.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2023-12-12T00:33:53.390 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-12T00:33:53.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-12T00:33:53.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T00:34:28.067 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2023-12-12T00:34:37.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-12T00:34:37.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T00:34:55.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2023-12-12T00:35:07.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-12T00:35:07.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T00:35:09.039 DEBUG:teuthology.orchestra.run.smithi162:> true 2023-12-12T00:35:09.323 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2023-12-12T00:35:09.323 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-527.el8.x86_64"... 2023-12-12T00:35:09.324 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2023-12-12T00:35:09.342 INFO:teuthology.orchestra.run.smithi162.stdout:4.18.0-527.el8.x86_64 2023-12-12T00:35:09.343 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-527.el8.x86_64 vs 4.18.0-527.el8.x86_64 2023-12-12T00:35:09.343 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-12T00:35:09.343 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-12T00:35:10.344 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-12T00:35:10.344 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-12T00:35:10.423 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2023-12-12T00:35:10.438 DEBUG:teuthology.parallel:result is None 2023-12-12T00:35:10.439 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-12T00:35:10.447 INFO:teuthology.task.internal:Creating test directory... 2023-12-12T00:35:10.447 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-12T00:35:10.464 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-12T00:35:10.546 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-12T00:35:10.577 INFO:teuthology.task.internal:Creating archive directory... 2023-12-12T00:35:10.577 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-12T00:35:10.600 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-12T00:35:10.608 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-12T00:35:10.608 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-12T00:35:10.670 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T00:35:10.680 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T00:35:10.681 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-12T00:35:10.688 INFO:teuthology.task.internal:Configuring sudo... 2023-12-12T00:35:10.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-12T00:35:10.750 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-12T00:35:10.762 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-12T00:35:10.762 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-12T00:35:10.807 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T00:35:10.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T00:35:10.937 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T00:35:11.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T00:35:11.067 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2023-12-12T00:35:11.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-12T00:35:11.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2023-12-12T00:35:11.250 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-12T00:35:11.322 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-12T00:35:11.330 INFO:teuthology.task.internal:Starting timer... 2023-12-12T00:35:11.330 INFO:teuthology.run_tasks:Running task pcp... 2023-12-12T00:35:11.408 INFO:teuthology.run_tasks:Running task selinux... 2023-12-12T00:35:11.452 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-12-12T00:35:11.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2023-12-12T00:35:11.538 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2023-12-12T00:35:11.543 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-12T00:35:11.543 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2023-12-12T00:35:11.569 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2023-12-12T00:35:11.569 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2023-12-12T00:35:11.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-12T00:35:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T00:35:11.639 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-12T00:35:11.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2023-12-12T00:35:11.706 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-12T00:35:11.717 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-12T00:35:11.717 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-12T00:35:11.735 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-12T00:35:11.736 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-12T00:41:06.603 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487624/ansible_failures.yaml 2023-12-12T00:41:06.606 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 500 - HTTP Error 500: INTERNAL SERVER ERROR', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-12T00:41:06.684 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8f9f79c864484ab393d0b018031978e0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 500 - HTTP Error 500: INTERNAL SERVER ERROR', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-12T00:41:06.686 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-12T00:41:06.695 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-12T00:41:06.696 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-12T00:41:06.719 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-12T00:41:06.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-12T00:41:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T00:41:06.852 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-12T00:41:06.861 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-12T00:41:06.876 INFO:teuthology.task.internal:Duration was 355.545082 seconds 2023-12-12T00:41:06.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-12T00:41:06.891 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-12T00:41:06.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-12T00:41:06.939 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-12T00:41:07.176 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-12T00:41:07.176 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2023-12-12T00:41:07.176 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-12T00:41:07.198 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-12T00:41:07.199 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-12T00:41:07.274 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-12T00:41:07.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-12T00:41:07.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-12T00:41:07.382 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-12T00:41:07.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-12T00:41:07.410 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-12T00:41:07.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-12T00:41:07.479 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2023-12-12T00:41:07.496 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-12T00:41:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T00:41:07.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-12T00:41:07.563 INFO:teuthology.task.internal:Transferring archived files... 2023-12-12T00:41:07.564 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487624/remote/smithi162 2023-12-12T00:41:07.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-12T00:41:07.647 INFO:teuthology.task.internal:Removing archive directory... 2023-12-12T00:41:07.648 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-12T00:41:07.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-12T00:41:07.711 INFO:teuthology.task.internal:Not uploading archives. 2023-12-12T00:41:07.711 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-12T00:41:07.721 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-12T00:41:07.721 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-12T00:41:07.758 INFO:teuthology.orchestra.run.smithi162.stdout: 525366 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 12 00:41 /home/ubuntu/cephtest 2023-12-12T00:41:07.759 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-12T00:41:07.768 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-12T00:41:07.795 INFO:teuthology.nuke:Checking targets against current locks 2023-12-12T00:41:07.824 DEBUG:teuthology.nuke:shortname: smithi162 2023-12-12T00:41:07.824 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-12T00:41:07.848 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487624', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 00:26:54.057352', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGDjZsN84llzgWCxN68NZGt8z00PwrXciT+xNGbTgUbvnID9Uaucgi9ggS34RLDrvkvewKEtCKaCaG4RTmF19jY='} 2023-12-12T00:41:07.893 INFO:teuthology.orchestra.console.smithi162:Power off 2023-12-12T00:41:07.894 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-12T00:41:07.918 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2023-12-12T00:41:07.919 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T00:41:07.942 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2023-12-12T00:41:11.943 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T00:41:12.053 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2023-12-12T00:41:16.054 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T00:41:16.164 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2023-12-12T00:41:20.169 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T00:41:20.279 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2023-12-12T00:41:20.380 INFO:teuthology.orchestra.console.smithi162:Power off completed 2023-12-12T00:41:20.489 INFO:teuthology.run:Summary data: description: rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} duration: 355.5450818538666 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 500 - HTTP Error 500: INTERNAL SERVER ERROR'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8f9f79c864484ab393d0b018031978e0 status: dead success: false 2023-12-12T00:41:20.490 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-12T00:41:20.596 INFO:teuthology.run:DEAD