2023-01-25T16:45:53.931 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-25T16:45:53.932 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-01-24_22:35:33-rbd-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137634 --description rbd/singleton/{all/permissions objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{centos_8}} --owner scheduled_yuriw@teuthology --name yuriw-2023-01-24_22:35:33-rbd-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-01-24_22:35:33-rbd-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137634/orig.config.yaml 2023-01-25T16:45:53.959 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-25T16:45:54.016 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_22:35:33-rbd-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137634 branch: wip-yuri6-testing-2023-01-24-0746 description: rbd/singleton/{all/permissions objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{centos_8}} email: yweinstein@radhat.com first_in_suite: false job_id: '7137634' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_22:35:33-rbd-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2023-01-24-0746 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2023-01-24-0746 sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - client.0 sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c sleep_before_teardown: 0 suite: rbd suite_branch: wip-yuri6-testing-2023-01-24-0746 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c targets: smithi012.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYO0NIfEZ3ZCIUq/fBp6fOHQMlrUCanTFCAIFnndcqu++wdIS/SXF9ToydKwmCYBh8IceEwxvPctCs9V1gK0xLIP4PQKZD/lNxIxJ4WfQV7G7D0GSnr8MRTBwmBnI0ikv2BwCnpS/Wicr6IXMDYx/RYAoa036had2j0Yh0Yblm9fsziwx/ZsDvYL08+0ZrwjxTVKetxZFAj2HRNpkYrF4VGsEsUs97H1/y/6hLvuQzq7lGR7LHF96bKQ1CmAYEeachd1iMProSngIaY7qpSIJJX9DWAidPL3LzIZjTYFfSqFdILy4oSrfIs2PBqFj08STeQOwB9GHgRy36lU605+msCq2W7QZtZGrRUsWZrbCmL/6Wx363pUVXwWPnn7uRSE/DIZlhReclukutTT2pTnB9HHFGmYI+vvT3cdv03809IwThXP9z7+nTLTm42WwkjsmGxCKkuBMlPQFtio3xNWsZ04CpJ6ew1sWoOqkxqVqIsySQKZYVJuEP3YjD2Rnjr2E= tasks: - install: null - ceph: fs: xfs - workunit: clients: all: - rbd/permissions.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_22:35:33 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-25T16:45:54.016 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa; will attempt to use it 2023-01-25T16:45:54.017 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa/tasks 2023-01-25T16:45:54.017 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-25T16:45:54.019 INFO:teuthology.task.internal:Checking packages... 2023-01-25T16:45:54.045 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '44e4095da12a8269eb6cfe163cbb0d00b1a6412c' 2023-01-25T16:45:54.046 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-25T16:45:54.046 INFO:teuthology.packaging:ref: None 2023-01-25T16:45:54.047 INFO:teuthology.packaging:tag: None 2023-01-25T16:45:54.047 INFO:teuthology.packaging:branch: wip-yuri6-testing-2023-01-24-0746 2023-01-25T16:45:54.047 INFO:teuthology.packaging:sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T16:45:54.048 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri6-testing-2023-01-24-0746 2023-01-25T16:45:54.367 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1986.g44e4095d 2023-01-25T16:45:54.368 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-25T16:45:54.376 INFO:teuthology.task.internal:no buildpackages task found 2023-01-25T16:45:54.376 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-25T16:45:54.401 INFO:teuthology.task.internal:Saving configuration 2023-01-25T16:45:54.412 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-25T16:45:54.419 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-25T16:45:54.443 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_22:35:33-rbd-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137634', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-25 16:41:28.470574', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDSHlYeyZQ0P558yo7+SdsGat2Oa/4rDvY4Z+RjP2keGt3vpt0KaqxgrILaHZL0oWwUiQygQc0U3Z2yCm0+fY54='} 2023-01-25T16:45:54.443 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-25T16:45:54.452 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'client.0'] 2023-01-25T16:45:54.453 INFO:teuthology.run_tasks:Running task console_log... 2023-01-25T16:45:54.491 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2b287185e0>, signals=[15]) 2023-01-25T16:45:54.492 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-25T16:45:54.500 INFO:teuthology.task.internal:Opening connections... 2023-01-25T16:45:54.501 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2023-01-25T16:45:54.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T16:45:54.572 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-25T16:45:54.583 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2023-01-25T16:45:54.648 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2023-01-25T16:45:54.648 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2023-01-25T16:45:54.707 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2023-01-25T16:45:54.707 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="8" 2023-01-25T16:45:54.707 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2023-01-25T16:45:54.707 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2023-01-25T16:45:54.708 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="8" 2023-01-25T16:45:54.708 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el8" 2023-01-25T16:45:54.708 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-25T16:45:54.708 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2023-01-25T16:45:54.709 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-25T16:45:54.709 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2023-01-25T16:45:54.709 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-25T16:45:54.709 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-25T16:45:54.709 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-25T16:45:54.710 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2023-01-25T16:45:54.734 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-25T16:45:54.745 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-25T16:45:54.754 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-25T16:45:54.754 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2023-01-25T16:45:54.773 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-25T16:45:54.782 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-25T16:45:54.782 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2023-01-25T16:45:54.839 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-25T16:45:54.881 INFO:teuthology.run_tasks:Running task kernel... 2023-01-25T16:45:54.901 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-25T16:45:54.901 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-25T16:45:54.902 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-25T16:45:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T16:45:54.921 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-01-25T16:45:54.979 INFO:teuthology.orchestra.run.smithi012.stdout:4.18.0-408.el8.x86_64 2023-01-25T16:45:54.980 INFO:teuthology.task.kernel:Running kernel on smithi012: 4.18.0-408.el8.x86_64 2023-01-25T16:45:54.980 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2023-01-25T16:45:58.548 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-8 - AppStream 36 MB/s | 27 MB 00:00 2023-01-25T16:46:04.690 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-8 - BaseOS 42 MB/s | 27 MB 00:00 2023-01-25T16:46:09.522 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-8 - Extras 158 kB/s | 18 kB 00:00 2023-01-25T16:46:09.949 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-8 - PowerTools 20 MB/s | 5.6 MB 00:00 2023-01-25T16:46:11.473 INFO:teuthology.orchestra.run.smithi012.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-01-25T16:46:13.115 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 13 MB 00:01 2023-01-25T16:46:16.599 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 582 kB/s | 24 kB 00:00 2023-01-25T16:46:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-25T16:46:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-25T16:46:19.996 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-01-25T16:46:19.996 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-01-25T16:46:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-01-25T16:46:19.997 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-01-25T16:46:19.997 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-01-25T16:46:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-25T16:46:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-25T16:46:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-25T16:46:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-01-25T16:46:19.998 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-01-25T16:46:19.998 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-01-25T16:46:19.998 INFO:teuthology.orchestra.run.smithi012.stdout:Install 3 Packages 2023-01-25T16:46:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-01-25T16:46:19.999 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 85 M 2023-01-25T16:46:19.999 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 94 M 2023-01-25T16:46:19.999 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-01-25T16:46:20.408 INFO:teuthology.orchestra.run.smithi012.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 22 MB/s | 9.2 MB 00:00 2023-01-25T16:46:21.108 INFO:teuthology.orchestra.run.smithi012.stdout:(2/3): kernel-core-4.18.0-448.el8.x86_64.rpm 37 MB/s | 41 MB 00:01 2023-01-25T16:46:21.525 INFO:teuthology.orchestra.run.smithi012.stdout:(3/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 22 MB/s | 34 MB 00:01 2023-01-25T16:46:21.528 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-01-25T16:46:21.528 INFO:teuthology.orchestra.run.smithi012.stdout:Total 55 MB/s | 85 MB 00:01 2023-01-25T16:46:22.195 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-01-25T16:46:22.624 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-01-25T16:46:22.624 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-01-25T16:46:24.590 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-01-25T16:46:24.622 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-01-25T16:46:27.507 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-01-25T16:46:28.071 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T16:46:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T16:46:32.504 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T16:46:36.053 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T16:46:36.315 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T16:46:59.586 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-25T16:46:59.613 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T16:47:00.067 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T16:47:00.067 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-25T16:47:00.068 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-25T16:47:00.702 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T16:47:00.703 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-01-25T16:47:00.703 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-01-25T16:47:00.703 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-25T16:47:00.703 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-25T16:47:00.704 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-01-25T16:47:00.704 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-01-25T16:47:00.822 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel --last | head -n 1 2023-01-25T16:47:01.025 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 04:46:36 PM UTC 2023-01-25T16:47:01.027 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T16:47:01.027 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-25T16:47:01.027 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-25T16:47:01.027 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-25T16:47:01.028 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-25T16:47:01.028 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2023-01-25T16:47:01.028 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel --last | head -n 1 2023-01-25T16:47:01.224 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 04:46:36 PM UTC 2023-01-25T16:47:01.226 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T16:47:01.227 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2023-01-25T16:47:01.261 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2023-01-25T16:47:01.261 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2023-01-25T16:47:01.261 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.02 2023-01-25T16:47:01.262 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 129.el8 2023-01-25T16:47:01.262 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2023-01-25T16:47:01.262 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-25T16:47:01.262 INFO:teuthology.orchestra.run.smithi012.stdout:Group : System Environment/Base 2023-01-25T16:47:01.262 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 9265586 2023-01-25T16:47:01.263 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2023-01-25T16:47:01.263 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-25T16:47:01.263 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-25T16:47:01.263 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-25T16:47:01.263 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-25T16:47:01.264 INFO:teuthology.orchestra.run.smithi012.stdout:Relocations : (not relocatable) 2023-01-25T16:47:01.264 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : CentOS BuildSystem 2023-01-25T16:47:01.264 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2023-01-25T16:47:01.264 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-25T16:47:01.264 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2023-01-25T16:47:01.265 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2023-01-25T16:47:01.265 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-01-25T16:47:01.265 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-25T16:47:01.265 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-25T16:47:01.265 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-25T16:47:01.266 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2023-01-25T16:47:01.266 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-01-25T16:47:01.266 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2023-01-25T16:47:01.267 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-25T16:47:01.267 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 4.18.0-448.el8.x86_64 2023-01-25T16:47:01.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-25T16:47:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2023-01-25T16:47:04.803 INFO:teuthology.orchestra.run.smithi012.stderr:done 2023-01-25T16:47:04.805 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-01-25T16:47:04.825 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.anQrnjBoUe 2023-01-25T16:47:04.826 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.anQrnjBoUe 2023-01-25T16:47:04.907 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.anQrnjBoUe 2023-01-25T16:47:05.057 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.anQrnjBoUe is 6KB 2023-01-25T16:47:05.108 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.anQrnjBoUe 2023-01-25T16:47:05.126 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2023-01-25T16:47:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-25T16:47:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-25T16:47:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-25T16:47:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-25T16:47:05.194 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-25T16:47:05.372 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2023-01-25T16:47:35.404 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-25T16:47:35.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-01-25T16:47:35.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T16:47:53.964 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2023-01-25T16:48:23.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-01-25T16:48:23.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T16:48:31.423 DEBUG:teuthology.orchestra.run.smithi012:> true 2023-01-25T16:48:31.762 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-01-25T16:48:31.762 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-448.el8.x86_64"... 2023-01-25T16:48:31.763 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-01-25T16:48:31.782 INFO:teuthology.orchestra.run.smithi012.stdout:4.18.0-448.el8.x86_64 2023-01-25T16:48:31.782 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-25T16:48:31.783 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-25T16:48:31.783 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-25T16:48:32.784 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-25T16:48:32.785 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-25T16:48:32.896 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2023-01-25T16:48:32.922 DEBUG:teuthology.parallel:result is None 2023-01-25T16:48:32.923 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-25T16:48:32.932 INFO:teuthology.task.internal:Creating test directory... 2023-01-25T16:48:32.933 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-25T16:48:32.954 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-25T16:48:32.965 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-25T16:48:32.979 INFO:teuthology.task.internal:Creating archive directory... 2023-01-25T16:48:32.980 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-25T16:48:33.028 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-25T16:48:33.039 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-25T16:48:33.039 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-25T16:48:33.122 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T16:48:33.134 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T16:48:33.136 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-25T16:48:33.147 INFO:teuthology.task.internal:Configuring sudo... 2023-01-25T16:48:33.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-25T16:48:33.177 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-25T16:48:33.188 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-25T16:48:33.188 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-25T16:48:33.236 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T16:48:33.329 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T16:48:33.372 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T16:48:33.466 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T16:48:33.502 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-01-25T16:48:33.503 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-25T16:48:33.573 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2023-01-25T16:48:33.672 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-25T16:48:34.972 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-25T16:48:34.983 INFO:teuthology.task.internal:Starting timer... 2023-01-25T16:48:34.983 INFO:teuthology.run_tasks:Running task pcp... 2023-01-25T16:48:35.000 INFO:teuthology.run_tasks:Running task selinux... 2023-01-25T16:48:35.014 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-25T16:48:35.014 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2023-01-25T16:48:35.091 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2023-01-25T16:48:35.093 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-25T16:48:35.093 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2023-01-25T16:48:35.138 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2023-01-25T16:48:35.138 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive'} 2023-01-25T16:48:35.139 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-25T16:48:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T16:48:35.208 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-25T16:48:35.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2023-01-25T16:48:35.292 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-25T16:48:35.302 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-01-25T16:48:35.308 INFO:teuthology.repo_utils:Fetching main from origin 2023-01-25T16:48:35.435 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-25T16:48:35.450 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-25T16:48:35.452 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-25T16:54:05.223 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-01-24_22:35:33-rbd-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137634/ansible_failures.yaml 2023-01-25T16:54:05.225 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/ansible.py", line 422, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/ansible.py", line 265, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/ansible.py", line 291, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/ansible.py", line 316, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: {'smithi012.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': 'ceph/python3-asyncssh', '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': ['dnf', '-y', 'copr', 'enable', 'ceph/python3-asyncssh'], 'delta': '0:00:09.853746', 'end': '2023-01-25 16:54:04.804770', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'dnf -y copr enable ceph/python3-asyncssh', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': 'ceph/python3-asyncssh', 'msg': 'non-zero return code', 'rc': 1, 'start': '2023-01-25 16:53:54.951024', 'stderr': 'Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection reset by peer', 'stderr_lines': ['Enabling a Copr repository. Please note that this repository is not part', 'of the main distribution, and quality may vary.', '', 'The Fedora Project does not exercise any power over the contents of', 'this repository beyond the rules outlined in the Copr FAQ at', ',', 'and packages are not held to any quality or security level.', '', 'Please do not file bug reports about these packages in Fedora', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection reset by peer'], 'stdout': '', 'stdout_lines': []}], 'warnings': ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]}} 2023-01-25T16:54:06.842 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ce0008a74c2c4d27a9fd500c3efb27b6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/ansible.py", line 422, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/ansible.py", line 265, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/ansible.py", line 291, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/task/ansible.py", line 316, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: {'smithi012.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': 'ceph/python3-asyncssh', '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': ['dnf', '-y', 'copr', 'enable', 'ceph/python3-asyncssh'], 'delta': '0:00:09.853746', 'end': '2023-01-25 16:54:04.804770', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'dnf -y copr enable ceph/python3-asyncssh', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': 'ceph/python3-asyncssh', 'msg': 'non-zero return code', 'rc': 1, 'start': '2023-01-25 16:53:54.951024', 'stderr': 'Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection reset by peer', 'stderr_lines': ['Enabling a Copr repository. Please note that this repository is not part', 'of the main distribution, and quality may vary.', '', 'The Fedora Project does not exercise any power over the contents of', 'this repository beyond the rules outlined in the Copr FAQ at', ',', 'and packages are not held to any quality or security level.', '', 'Please do not file bug reports about these packages in Fedora', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection reset by peer'], 'stdout': '', 'stdout_lines': []}], 'warnings': ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]}} 2023-01-25T16:54:06.849 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-01-25T16:54:06.857 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-01-25T16:54:06.857 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-01-25T16:54:06.881 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-01-25T16:54:06.982 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-25T16:54:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T16:54:07.017 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-01-25T16:54:07.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-01-25T16:54:07.060 INFO:teuthology.task.internal:Duration was 332.076346 seconds 2023-01-25T16:54:07.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-01-25T16:54:07.096 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-01-25T16:54:07.096 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-01-25T16:54:07.147 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-25T16:54:07.394 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-01-25T16:54:07.394 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2023-01-25T16:54:07.395 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-01-25T16:54:07.419 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-01-25T16:54:07.419 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-25T16:54:07.499 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-01-25T16:54:07.500 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-01-25T16:54:07.602 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-01-25T16:54:07.610 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-01-25T16:54:07.611 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-01-25T16:54:07.638 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-01-25T16:54:07.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-01-25T16:54:07.705 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2023-01-25T16:54:07.719 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2023-01-25T16:54:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T16:54:07.777 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-01-25T16:54:07.800 INFO:teuthology.task.internal:Transferring archived files... 2023-01-25T16:54:07.801 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-01-24_22:35:33-rbd-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137634/remote/smithi012 2023-01-25T16:54:07.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2023-01-25T16:54:07.879 INFO:teuthology.task.internal:Removing archive directory... 2023-01-25T16:54:07.880 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2023-01-25T16:54:07.899 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-01-25T16:54:07.908 INFO:teuthology.task.internal:Not uploading archives. 2023-01-25T16:54:07.909 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-01-25T16:54:07.922 INFO:teuthology.task.internal:Tidying up after the test... 2023-01-25T16:54:07.923 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-01-25T16:54:07.957 INFO:teuthology.orchestra.run.smithi012.stdout: 525248 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 25 16:54 /home/ubuntu/cephtest 2023-01-25T16:54:07.958 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-01-25T16:54:07.981 INFO:teuthology.nuke:Checking targets against current locks 2023-01-25T16:54:08.009 DEBUG:teuthology.nuke:shortname: smithi012 2023-01-25T16:54:08.009 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-25T16:54:08.033 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_22:35:33-rbd-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137634', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-25 16:41:28.470574', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYO0NIfEZ3ZCIUq/fBp6fOHQMlrUCanTFCAIFnndcqu++wdIS/SXF9ToydKwmCYBh8IceEwxvPctCs9V1gK0xLIP4PQKZD/lNxIxJ4WfQV7G7D0GSnr8MRTBwmBnI0ikv2BwCnpS/Wicr6IXMDYx/RYAoa036had2j0Yh0Yblm9fsziwx/ZsDvYL08+0ZrwjxTVKetxZFAj2HRNpkYrF4VGsEsUs97H1/y/6hLvuQzq7lGR7LHF96bKQ1CmAYEeachd1iMProSngIaY7qpSIJJX9DWAidPL3LzIZjTYFfSqFdILy4oSrfIs2PBqFj08STeQOwB9GHgRy36lU605+msCq2W7QZtZGrRUsWZrbCmL/6Wx363pUVXwWPnn7uRSE/DIZlhReclukutTT2pTnB9HHFGmYI+vvT3cdv03809IwThXP9z7+nTLTm42WwkjsmGxCKkuBMlPQFtio3xNWsZ04CpJ6ew1sWoOqkxqVqIsySQKZYVJuEP3YjD2Rnjr2E='} 2023-01-25T16:54:08.083 INFO:teuthology.orchestra.console:Power off smithi012 2023-01-25T16:54:08.084 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-01-25T16:54:08.105 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-01-25T16:54:12.127 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-01-25T16:54:16.240 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-01-25T16:54:20.352 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-01-25T16:54:20.562 INFO:teuthology.orchestra.console:Power off for smithi012 completed 2023-01-25T16:54:20.672 INFO:teuthology.run:Summary data: description: rbd/singleton/{all/permissions objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{centos_8}} duration: 332.07634592056274 failure_reason: '{''smithi012.front.sepia.ceph.com'': {''changed'': True, ''msg'': ''All items completed'', ''results'': [{''_ansible_item_label'': ''ceph/python3-asyncssh'', ''_ansible_no_log'': False, ''ansible_loop_var'': ''item'', ''changed'': True, ''cmd'': [''dnf'', ''-y'', ''copr'', ''enable'', ''ceph/python3-asyncssh''], ''delta'': ''0:00:09.853746'', ''end'': ''2023-01-25 16:54:04.804770'', ''failed'': True, ''invocation'': {''module_args'': {''_raw_params'': ''dnf -y copr enable ceph/python3-asyncssh'', ''_uses_shell'': False, ''argv'': None, ''chdir'': None, ''creates'': None, ''executable'': None, ''removes'': None, ''stdin'': None, ''stdin_add_newline'': True, ''strip_empty_ends'': True, ''warn'': True}}, ''item'': ''ceph/python3-asyncssh'', ''msg'': ''non-zero return code'', ''rc'': 1, ''start'': ''2023-01-25 16:53:54.951024'', ''stderr'': ''Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection reset by peer'', ''stderr_lines'': [''Enabling a Copr repository. Please note that this repository is not part'', ''of the main distribution, and quality may vary.'', '''', ''The Fedora Project does not exercise any power over the contents of'', ''this repository beyond the rules outlined in the Copr FAQ at'', '','', ''and packages are not held to any quality or security level.'', '''', ''Please do not file bug reports about these packages in Fedora'', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection reset by peer''], ''stdout'': '''', ''stdout_lines'': []}], ''warnings'': ["Consider using the dnf module rather than running ''dnf''. If you need to use command because dnf is insufficient you can add ''warn: false'' to this command task or set ''command_warnings=False'' in ansible.cfg to get rid of this message."]}}' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ce0008a74c2c4d27a9fd500c3efb27b6 status: dead success: false 2023-01-25T16:54:20.673 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-25T16:54:20.737 INFO:teuthology.run:DEAD