2023-10-18T14:28:41.279 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-18T14:28:41.280 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546 --name yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi --description smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_cli_import_export}} --verbose -- /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546/orig.config.yaml 2023-10-18T14:28:41.310 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:28:41.379 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546 branch: quincy-release description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_cli_import_export}} email: null first_in_suite: false job_id: '7431546' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: quincy-release ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-quincy-test-fixes sha1: 69fb3d4792adb6f1b63824c294e62c636f9a850d owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mds.a - mgr.x - osd.0 - osd.1 - - mon.b - mds.b - mon.c - mgr.y - osd.2 - osd.3 - - client.0 seed: 7103 sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 sleep_before_teardown: 0 subset: 111/120000 suite: smoke suite_branch: wip-quincy-test-fixes suite_path: /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa suite_relpath: qa suite_repo: https://github.com/pdvian/ceph.git suite_sha1: 69fb3d4792adb6f1b63824c294e62c636f9a850d targets: smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBoQdW73cZxv1fom3GBWWVhUCxE9Oarq0oP42ywimqxZkh3NoCCHKiDLO2TUJXT4Zhyr3WS5SYEuPVHHgd/2Hcc= smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNjhzcZjIyDK+DJtLZSLYBmd53yrAiCerwyGLgyfsCkCtoh6jPicrhINtSaU8kBZTGmwjpKRzj9GSxDCgMGy4+Y= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTKe4JOv6NNM33khpORUsC0CVo2m+62lOt5hfn4XcolFnZHXCNKks3eOGVw1nRAB3oN913vi9Pz+8yg3rEXuNI= tasks: - install: cleanup: true - ceph: fs: xfs log-ignorelist: - \(POOL_APP_NOT_ENABLED\) - ceph-fuse: null - workunit: clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-18_13:55:44 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T14:28:41.379 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa; will attempt to use it 2023-10-18T14:28:41.380 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa/tasks 2023-10-18T14:28:41.380 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T14:28:41.382 INFO:teuthology.task.internal:Checking packages... 2023-10-18T14:28:41.404 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9d6d32bb3452d5179bde2ee1cfa05df7a65f4586' 2023-10-18T14:28:41.404 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T14:28:41.405 INFO:teuthology.packaging:ref: None 2023-10-18T14:28:41.405 INFO:teuthology.packaging:tag: None 2023-10-18T14:28:41.405 INFO:teuthology.packaging:branch: quincy-release 2023-10-18T14:28:41.406 INFO:teuthology.packaging:sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 2023-10-18T14:28:41.406 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy-release 2023-10-18T14:28:41.581 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1415.g9d6d32bb 2023-10-18T14:28:41.581 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T14:28:41.587 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T14:28:41.588 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T14:28:41.593 INFO:teuthology.task.internal:Saving configuration 2023-10-18T14:28:41.605 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T14:28:41.613 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:28:41.638 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:23:25.637787', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBoQdW73cZxv1fom3GBWWVhUCxE9Oarq0oP42ywimqxZkh3NoCCHKiDLO2TUJXT4Zhyr3WS5SYEuPVHHgd/2Hcc='} 2023-10-18T14:28:41.663 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:23:25.639188', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNjhzcZjIyDK+DJtLZSLYBmd53yrAiCerwyGLgyfsCkCtoh6jPicrhINtSaU8kBZTGmwjpKRzj9GSxDCgMGy4+Y='} 2023-10-18T14:28:41.684 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:23:25.636107', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTKe4JOv6NNM33khpORUsC0CVo2m+62lOt5hfn4XcolFnZHXCNKks3eOGVw1nRAB3oN913vi9Pz+8yg3rEXuNI='} 2023-10-18T14:28:41.684 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T14:28:41.691 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-10-18T14:28:41.691 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-10-18T14:28:41.691 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['client.0'] 2023-10-18T14:28:41.691 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T14:28:41.797 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4125e7fa60>, signals=[15]) 2023-10-18T14:28:41.798 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T14:28:41.805 INFO:teuthology.task.internal:Opening connections... 2023-10-18T14:28:41.805 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2023-10-18T14:28:41.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:28:41.875 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2023-10-18T14:28:41.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:28:41.944 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2023-10-18T14:28:41.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:28:42.014 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T14:28:42.021 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2023-10-18T14:28:42.044 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2023-10-18T14:28:42.044 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2023-10-18T14:28:42.104 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2023-10-18T14:28:42.104 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="8" 2023-10-18T14:28:42.105 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2023-10-18T14:28:42.105 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2023-10-18T14:28:42.105 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="8" 2023-10-18T14:28:42.105 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:28:42.105 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-18T14:28:42.105 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2023-10-18T14:28:42.106 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-18T14:28:42.106 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2023-10-18T14:28:42.106 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:28:42.106 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:28:42.106 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-18T14:28:42.107 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2023-10-18T14:28:42.125 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2023-10-18T14:28:42.147 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2023-10-18T14:28:42.148 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2023-10-18T14:28:42.209 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2023-10-18T14:28:42.209 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8" 2023-10-18T14:28:42.209 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2023-10-18T14:28:42.209 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2023-10-18T14:28:42.210 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8" 2023-10-18T14:28:42.210 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:28:42.210 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-18T14:28:42.210 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2023-10-18T14:28:42.210 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-18T14:28:42.211 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2023-10-18T14:28:42.211 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:28:42.211 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:28:42.211 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-18T14:28:42.211 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2023-10-18T14:28:42.234 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2023-10-18T14:28:42.256 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2023-10-18T14:28:42.257 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2023-10-18T14:28:42.317 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2023-10-18T14:28:42.318 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8" 2023-10-18T14:28:42.318 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2023-10-18T14:28:42.318 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2023-10-18T14:28:42.318 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8" 2023-10-18T14:28:42.319 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:28:42.319 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-18T14:28:42.319 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2023-10-18T14:28:42.319 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-18T14:28:42.320 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2023-10-18T14:28:42.320 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:28:42.320 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:28:42.320 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-18T14:28:42.320 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2023-10-18T14:28:42.342 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T14:28:42.351 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T14:28:42.357 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T14:28:42.357 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:28:42.359 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:28:42.362 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:28:42.383 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T14:28:42.389 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T14:28:42.389 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:28:42.420 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:28:42.423 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:28:42.455 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T14:28:42.534 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T14:28:42.547 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-18T14:28:42.548 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-18T14:28:42.548 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:28:42.549 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:28:42.549 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:28:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:28:42.570 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2023-10-18T14:28:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:28:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:28:42.572 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2023-10-18T14:28:42.572 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2023-10-18T14:28:42.631 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-489.el8.x86_64 2023-10-18T14:28:42.632 INFO:teuthology.task.kernel:Running kernel on smithi099: 4.18.0-489.el8.x86_64 2023-10-18T14:28:42.632 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2023-10-18T14:28:42.633 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-489.el8.x86_64 2023-10-18T14:28:42.634 INFO:teuthology.task.kernel:Running kernel on smithi123: 4.18.0-489.el8.x86_64 2023-10-18T14:28:42.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum install -y kernel 2023-10-18T14:28:42.635 INFO:teuthology.orchestra.run.smithi081.stdout:4.18.0-489.el8.x86_64 2023-10-18T14:28:42.635 INFO:teuthology.task.kernel:Running kernel on smithi081: 4.18.0-489.el8.x86_64 2023-10-18T14:28:42.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2023-10-18T14:28:48.475 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 10 MB/s | 33 MB 00:03 2023-10-18T14:28:48.500 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - AppStream 10 MB/s | 33 MB 00:03 2023-10-18T14:28:48.547 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - AppStream 10 MB/s | 33 MB 00:03 2023-10-18T14:28:56.908 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - BaseOS 21 MB/s | 50 MB 00:02 2023-10-18T14:28:57.597 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 17 MB/s | 50 MB 00:02 2023-10-18T14:28:57.805 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - BaseOS 16 MB/s | 50 MB 00:03 2023-10-18T14:29:04.458 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - Extras 59 kB/s | 18 kB 00:00 2023-10-18T14:29:05.122 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 58 kB/s | 18 kB 00:00 2023-10-18T14:29:05.681 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-10-18T14:29:05.959 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-10-18T14:29:06.429 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras 12 kB/s | 18 kB 00:01 2023-10-18T14:29:06.769 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 24 MB/s | 16 MB 00:00 2023-10-18T14:29:07.571 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras common packages 7.2 kB/s | 6.9 kB 00:00 2023-10-18T14:29:07.848 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-10-18T14:29:08.600 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - Extras common packages 1.8 kB/s | 6.9 kB 00:03 2023-10-18T14:29:08.952 INFO:teuthology.orchestra.run.smithi081.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-10-18T14:29:09.015 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2023-10-18T14:29:10.252 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-18T14:29:10.560 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 415 kB/s | 24 kB 00:00 2023-10-18T14:29:12.856 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 328 kB/s | 24 kB 00:00 2023-10-18T14:29:14.033 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-18T14:29:14.033 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-18T14:29:14.044 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 483 kB/s | 24 kB 00:00 2023-10-18T14:29:14.301 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-10-18T14:29:14.301 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-10-18T14:29:14.302 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-10-18T14:29:14.302 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-10-18T14:29:14.302 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-10-18T14:29:14.302 INFO:teuthology.orchestra.run.smithi123.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-18T14:29:14.303 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-10-18T14:29:14.303 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-18T14:29:14.303 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-18T14:29:14.303 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-10-18T14:29:14.303 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-10-18T14:29:14.304 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-10-18T14:29:14.304 INFO:teuthology.orchestra.run.smithi123.stdout:Install 3 Packages 2023-10-18T14:29:14.304 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-10-18T14:29:14.305 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 89 M 2023-10-18T14:29:14.305 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 96 M 2023-10-18T14:29:14.305 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-10-18T14:29:15.037 INFO:teuthology.orchestra.run.smithi123.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 15 MB/s | 10 MB 00:00 2023-10-18T14:29:16.389 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-18T14:29:16.389 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-18T14:29:16.635 INFO:teuthology.orchestra.run.smithi123.stdout:(2/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2023-10-18T14:29:16.643 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-10-18T14:29:16.644 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-10-18T14:29:16.644 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2023-10-18T14:29:16.645 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-10-18T14:29:16.645 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-10-18T14:29:16.645 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-18T14:29:16.645 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-10-18T14:29:16.645 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-18T14:29:16.646 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-18T14:29:16.646 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-10-18T14:29:16.646 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-10-18T14:29:16.646 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-10-18T14:29:16.646 INFO:teuthology.orchestra.run.smithi099.stdout:Install 3 Packages 2023-10-18T14:29:16.647 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-10-18T14:29:16.647 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 89 M 2023-10-18T14:29:16.647 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 96 M 2023-10-18T14:29:16.648 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-10-18T14:29:17.035 INFO:teuthology.orchestra.run.smithi123.stdout:(3/3): kernel-core-4.18.0-517.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2023-10-18T14:29:17.036 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-10-18T14:29:17.036 INFO:teuthology.orchestra.run.smithi123.stdout:Total 32 MB/s | 89 MB 00:02 2023-10-18T14:29:17.569 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-18T14:29:17.569 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-18T14:29:17.707 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-10-18T14:29:17.830 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-10-18T14:29:17.832 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-10-18T14:29:17.832 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-10-18T14:29:17.832 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-10-18T14:29:17.833 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-10-18T14:29:17.833 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-18T14:29:17.833 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-10-18T14:29:17.833 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-18T14:29:17.833 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-18T14:29:17.834 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-18T14:29:17.834 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-10-18T14:29:17.834 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-10-18T14:29:17.834 INFO:teuthology.orchestra.run.smithi081.stdout:Install 3 Packages 2023-10-18T14:29:17.834 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-18T14:29:17.835 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 89 M 2023-10-18T14:29:17.835 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 96 M 2023-10-18T14:29:17.835 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-10-18T14:29:17.852 INFO:teuthology.orchestra.run.smithi099.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 9.9 MB/s | 10 MB 00:01 2023-10-18T14:29:18.116 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-10-18T14:29:18.117 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-10-18T14:29:19.348 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-10-18T14:29:19.370 INFO:teuthology.orchestra.run.smithi081.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 6.7 MB/s | 10 MB 00:01 2023-10-18T14:29:19.371 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-10-18T14:29:20.407 INFO:teuthology.orchestra.run.smithi099.stdout:(2/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 10 MB/s | 36 MB 00:03 2023-10-18T14:29:22.004 INFO:teuthology.orchestra.run.smithi081.stdout:(2/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 8.7 MB/s | 36 MB 00:04 2023-10-18T14:29:22.134 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-10-18T14:29:22.291 INFO:teuthology.orchestra.run.smithi099.stdout:(3/3): kernel-core-4.18.0-517.el8.x86_64.rpm 7.9 MB/s | 43 MB 00:05 2023-10-18T14:29:22.291 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-10-18T14:29:22.292 INFO:teuthology.orchestra.run.smithi099.stdout:Total 16 MB/s | 89 MB 00:05 2023-10-18T14:29:22.562 INFO:teuthology.orchestra.run.smithi081.stdout:(3/3): kernel-core-4.18.0-517.el8.x86_64.rpm 9.2 MB/s | 43 MB 00:04 2023-10-18T14:29:22.563 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-10-18T14:29:22.563 INFO:teuthology.orchestra.run.smithi081.stdout:Total 19 MB/s | 89 MB 00:04 2023-10-18T14:29:22.682 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:29:22.967 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-10-18T14:29:23.237 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-10-18T14:29:23.395 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-10-18T14:29:23.395 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-10-18T14:29:23.667 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-10-18T14:29:23.667 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-10-18T14:29:24.603 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-10-18T14:29:24.627 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-10-18T14:29:24.892 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-10-18T14:29:24.913 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-10-18T14:29:26.363 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:29:26.781 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:29:27.387 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-10-18T14:29:27.682 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-10-18T14:29:27.996 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:29:28.234 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:29:30.153 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:29:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:31.591 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:29:31.806 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:29:31.995 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:29:32.192 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:29:35.380 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:29:35.574 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:29:35.675 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:35.864 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:52.034 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:52.058 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:29:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:29:53.324 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:53.325 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-10-18T14:29:53.325 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-10-18T14:29:53.325 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-18T14:29:53.326 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-18T14:29:53.326 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-10-18T14:29:53.326 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-10-18T14:29:53.465 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:29:53.699 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:29:53.699 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:29:53.700 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-18T14:29:53.700 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:29:53.700 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:29:53.700 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-18T14:29:53.700 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:29:53.701 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:29:53.909 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:29:53.909 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:29:53.909 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2023-10-18T14:29:53.943 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2023-10-18T14:29:53.943 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2023-10-18T14:29:53.944 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.02 2023-10-18T14:29:53.944 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 129.el8 2023-10-18T14:29:53.944 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2023-10-18T14:29:53.944 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-18T14:29:53.945 INFO:teuthology.orchestra.run.smithi123.stdout:Group : System Environment/Base 2023-10-18T14:29:53.945 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 9265586 2023-10-18T14:29:53.945 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2023-10-18T14:29:53.945 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-18T14:29:53.945 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-18T14:29:53.946 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-18T14:29:53.946 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-18T14:29:53.946 INFO:teuthology.orchestra.run.smithi123.stdout:Relocations : (not relocatable) 2023-10-18T14:29:53.946 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : CentOS BuildSystem 2023-10-18T14:29:53.946 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2023-10-18T14:29:53.947 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:29:53.947 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2023-10-18T14:29:53.947 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2023-10-18T14:29:53.947 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-10-18T14:29:53.947 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:29:53.948 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:29:53.948 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:29:53.948 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2023-10-18T14:29:53.948 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-10-18T14:29:53.948 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:29:53.949 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:29:53.950 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 4.18.0-517.el8.x86_64 2023-10-18T14:29:53.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:29:55.480 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2023-10-18T14:29:57.373 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:57.398 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:57.469 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:57.492 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:57.747 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:57.747 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:29:57.748 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:29:57.914 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:57.915 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:29:57.915 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:29:58.200 INFO:teuthology.orchestra.run.smithi123.stderr:done 2023-10-18T14:29:58.202 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2023-10-18T14:29:58.226 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.CRuzDz9bhy 2023-10-18T14:29:58.227 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CRuzDz9bhy 2023-10-18T14:29:58.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.CRuzDz9bhy 2023-10-18T14:29:58.424 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.CRuzDz9bhy is 6KB 2023-10-18T14:29:58.474 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.CRuzDz9bhy 2023-10-18T14:29:58.489 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:58.489 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-10-18T14:29:58.489 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-10-18T14:29:58.489 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-18T14:29:58.490 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-18T14:29:58.490 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-10-18T14:29:58.490 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-10-18T14:29:58.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:29:58.568 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-18T14:29:58.568 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-18T14:29:58.568 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-18T14:29:58.568 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-18T14:29:58.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-18T14:29:58.638 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:29:58.736 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2023-10-18T14:29:58.779 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:29:58.779 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-18T14:29:58.779 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-10-18T14:29:58.780 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-18T14:29:58.780 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-18T14:29:58.780 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-18T14:29:58.780 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-10-18T14:29:58.850 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:29:58.851 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:29:58.851 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-18T14:29:58.851 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:29:58.851 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:29:58.852 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-18T14:29:58.852 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi099.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:29:58.852 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:29:58.936 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:29:59.060 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:29:59.060 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:29:59.060 DEBUG:teuthology.orchestra.run.smithi099:> sudo rpm -qi grub2-tools 2023-10-18T14:29:59.094 INFO:teuthology.orchestra.run.smithi099.stdout:Name : grub2-tools 2023-10-18T14:29:59.094 INFO:teuthology.orchestra.run.smithi099.stdout:Epoch : 1 2023-10-18T14:29:59.095 INFO:teuthology.orchestra.run.smithi099.stdout:Version : 2.02 2023-10-18T14:29:59.095 INFO:teuthology.orchestra.run.smithi099.stdout:Release : 129.el8 2023-10-18T14:29:59.095 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: x86_64 2023-10-18T14:29:59.095 INFO:teuthology.orchestra.run.smithi099.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-18T14:29:59.095 INFO:teuthology.orchestra.run.smithi099.stdout:Group : System Environment/Base 2023-10-18T14:29:59.096 INFO:teuthology.orchestra.run.smithi099.stdout:Size : 9265586 2023-10-18T14:29:59.096 INFO:teuthology.orchestra.run.smithi099.stdout:License : GPLv3+ 2023-10-18T14:29:59.096 INFO:teuthology.orchestra.run.smithi099.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-18T14:29:59.096 INFO:teuthology.orchestra.run.smithi099.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-18T14:29:59.096 INFO:teuthology.orchestra.run.smithi099.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-18T14:29:59.097 INFO:teuthology.orchestra.run.smithi099.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-18T14:29:59.097 INFO:teuthology.orchestra.run.smithi099.stdout:Relocations : (not relocatable) 2023-10-18T14:29:59.097 INFO:teuthology.orchestra.run.smithi099.stdout:Packager : CentOS BuildSystem 2023-10-18T14:29:59.097 INFO:teuthology.orchestra.run.smithi099.stdout:Vendor : CentOS 2023-10-18T14:29:59.097 INFO:teuthology.orchestra.run.smithi099.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:29:59.097 INFO:teuthology.orchestra.run.smithi099.stdout:Summary : Support tools for GRUB. 2023-10-18T14:29:59.098 INFO:teuthology.orchestra.run.smithi099.stdout:Description : 2023-10-18T14:29:59.098 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-10-18T14:29:59.098 INFO:teuthology.orchestra.run.smithi099.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:29:59.099 INFO:teuthology.orchestra.run.smithi099.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:29:59.099 INFO:teuthology.orchestra.run.smithi099.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:29:59.099 INFO:teuthology.orchestra.run.smithi099.stdout:hardware devices. 2023-10-18T14:29:59.099 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-10-18T14:29:59.099 INFO:teuthology.orchestra.run.smithi099.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:29:59.100 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:29:59.101 INFO:teuthology.task.kernel:Updating grub on smithi099 to boot 4.18.0-517.el8.x86_64 2023-10-18T14:29:59.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:29:59.153 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:29:59.154 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:29:59.154 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-18T14:29:59.154 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:29:59.155 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:29:59.155 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-18T14:29:59.155 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:29:59.155 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:29:59.362 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:29:59.363 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:29:59.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2023-10-18T14:29:59.397 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2023-10-18T14:29:59.397 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2023-10-18T14:29:59.397 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.02 2023-10-18T14:29:59.397 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 129.el8 2023-10-18T14:29:59.398 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2023-10-18T14:29:59.398 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-18T14:29:59.398 INFO:teuthology.orchestra.run.smithi081.stdout:Group : System Environment/Base 2023-10-18T14:29:59.398 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 9265586 2023-10-18T14:29:59.399 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2023-10-18T14:29:59.399 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-18T14:29:59.399 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-18T14:29:59.399 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-18T14:29:59.399 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-18T14:29:59.400 INFO:teuthology.orchestra.run.smithi081.stdout:Relocations : (not relocatable) 2023-10-18T14:29:59.400 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : CentOS BuildSystem 2023-10-18T14:29:59.400 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2023-10-18T14:29:59.400 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:29:59.401 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2023-10-18T14:29:59.401 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2023-10-18T14:29:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-18T14:29:59.401 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:29:59.402 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:29:59.402 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:29:59.402 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2023-10-18T14:29:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-18T14:29:59.402 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:29:59.403 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:29:59.404 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 4.18.0-517.el8.x86_64 2023-10-18T14:29:59.404 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:30:00.839 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2023-10-18T14:30:00.987 INFO:teuthology.orchestra.run.smithi099.stderr:Generating grub configuration file ... 2023-10-18T14:30:02.938 INFO:teuthology.orchestra.run.smithi081.stderr:done 2023-10-18T14:30:02.940 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2023-10-18T14:30:02.976 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.DyfWoOWk9V 2023-10-18T14:30:02.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DyfWoOWk9V 2023-10-18T14:30:03.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.DyfWoOWk9V 2023-10-18T14:30:03.193 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.DyfWoOWk9V is 6KB 2023-10-18T14:30:03.274 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.DyfWoOWk9V 2023-10-18T14:30:03.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:30:03.372 INFO:teuthology.orchestra.run.smithi081.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-18T14:30:03.373 INFO:teuthology.orchestra.run.smithi081.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-18T14:30:03.373 INFO:teuthology.orchestra.run.smithi081.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-18T14:30:03.373 INFO:teuthology.orchestra.run.smithi081.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-18T14:30:03.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-18T14:30:03.380 INFO:teuthology.orchestra.run.smithi099.stderr:done 2023-10-18T14:30:03.382 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2023-10-18T14:30:03.416 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.l8fOBsonRF 2023-10-18T14:30:03.416 DEBUG:teuthology.orchestra.run.smithi099:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.l8fOBsonRF 2023-10-18T14:30:03.494 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.l8fOBsonRF 2023-10-18T14:30:03.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2023-10-18T14:30:03.632 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.l8fOBsonRF is 6KB 2023-10-18T14:30:03.658 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.l8fOBsonRF 2023-10-18T14:30:03.677 DEBUG:teuthology.orchestra.run.smithi099:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:30:03.770 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-18T14:30:03.771 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-18T14:30:03.772 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-18T14:30:03.772 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-18T14:30:03.774 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-18T14:30:03.888 DEBUG:teuthology.orchestra.run.smithi099:> sudo shutdown -r now 2023-10-18T14:30:28.761 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-18T14:30:28.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-10-18T14:30:28.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:30:33.579 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-18T14:30:33.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-18T14:30:33.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:30:33.892 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-18T14:30:33.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2023-10-18T14:30:33.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:30:47.162 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2023-10-18T14:30:52.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2023-10-18T14:30:52.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2023-10-18T14:30:56.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-10-18T14:30:56.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:30:59.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2023-10-18T14:31:01.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-18T14:31:01.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:01.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2023-10-18T14:31:01.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:04.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2023-10-18T14:31:04.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2023-10-18T14:31:11.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-10-18T14:31:11.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:16.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-18T14:31:16.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:16.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2023-10-18T14:31:16.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:34.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2023-10-18T14:31:34.778 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2023-10-18T14:31:43.455 DEBUG:teuthology.orchestra.run.smithi123:> true 2023-10-18T14:31:44.453 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-10-18T14:31:44.454 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-517.el8.x86_64"... 2023-10-18T14:31:44.454 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2023-10-18T14:31:44.471 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-517.el8.x86_64 2023-10-18T14:31:44.472 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-18T14:31:44.472 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:31:44.472 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-18T14:31:45.474 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2023-10-18T14:31:45.474 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:31:45.540 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2023-10-18T14:31:45.552 DEBUG:teuthology.parallel:result is None 2023-10-18T14:31:49.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-18T14:31:49.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:49.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2023-10-18T14:31:49.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:50.150 DEBUG:teuthology.orchestra.run.smithi081:> true 2023-10-18T14:31:50.203 DEBUG:teuthology.orchestra.run.smithi099:> true 2023-10-18T14:31:50.805 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2023-10-18T14:31:50.805 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-517.el8.x86_64"... 2023-10-18T14:31:50.806 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2023-10-18T14:31:50.828 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-517.el8.x86_64 2023-10-18T14:31:50.829 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-18T14:31:50.829 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:31:50.829 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-18T14:31:51.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-18T14:31:51.145 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-517.el8.x86_64"... 2023-10-18T14:31:51.145 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2023-10-18T14:31:51.542 INFO:teuthology.orchestra.run.smithi081.stdout:4.18.0-517.el8.x86_64 2023-10-18T14:31:51.542 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-18T14:31:51.542 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:31:51.543 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-18T14:31:51.831 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-18T14:31:51.831 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:31:52.367 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2023-10-18T14:31:52.378 DEBUG:teuthology.parallel:result is None 2023-10-18T14:31:52.544 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-18T14:31:52.544 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:31:52.616 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2023-10-18T14:31:52.712 DEBUG:teuthology.parallel:result is None 2023-10-18T14:31:52.713 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T14:31:52.719 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T14:31:52.720 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:31:52.722 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:31:52.725 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:31:52.747 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T14:31:52.754 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T14:31:52.762 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T14:31:52.762 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:31:52.781 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:31:52.785 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:31:52.829 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T14:31:52.836 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T14:31:52.837 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:31:52.858 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:31:52.860 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:31:52.902 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:31:52.904 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:31:52.913 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:31:52.915 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:31:52.935 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:31:52.946 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:31:52.948 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T14:31:52.955 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T14:31:52.956 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:31:52.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:31:52.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:31:52.991 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T14:31:53.000 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T14:31:53.000 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:31:53.026 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:31:53.028 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:31:53.051 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:31:53.126 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:31:53.183 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:31:53.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:31:53.310 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-18T14:31:53.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:31:53.377 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:31:53.423 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:31:53.474 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:31:53.559 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:31:53.601 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-10-18T14:31:53.601 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:31:53.668 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:31:53.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:31:53.774 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:31:53.862 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:31:53.911 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-10-18T14:31:53.911 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:31:53.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2023-10-18T14:31:53.986 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2023-10-18T14:31:53.989 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2023-10-18T14:31:54.040 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:31:54.045 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:31:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:31:54.566 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T14:31:54.574 INFO:teuthology.task.internal:Starting timer... 2023-10-18T14:31:54.574 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T14:31:54.592 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T14:31:54.599 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T14:31:54.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2023-10-18T14:31:54.696 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2023-10-18T14:31:54.698 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2023-10-18T14:31:54.778 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2023-10-18T14:31:54.779 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2023-10-18T14:31:54.856 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2023-10-18T14:31:54.857 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T14:31:54.858 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2023-10-18T14:31:54.905 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2023-10-18T14:31:54.906 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2023-10-18T14:31:54.934 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2023-10-18T14:31:54.935 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2023-10-18T14:31:54.964 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2023-10-18T14:31:54.964 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive'} 2023-10-18T14:31:54.965 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:31:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:31:54.993 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:31:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:31:55.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:31:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:31:55.049 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T14:31:55.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:31:55.082 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:31:55.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:31:55.138 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T14:31:55.148 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-18T14:31:55.149 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T14:31:55.165 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T14:31:55.166 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi099.front.sepia.ceph.com,smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T14:51:28.981 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546/ansible_failures.yaml 2023-10-18T14:51:28.983 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 504 - HTTP Error 504: Gateway Timeout', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:51:29.059 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5a9c0082ce154f02aa3f1bf155db5373 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 504 - HTTP Error 504: Gateway Timeout', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:51:29.061 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T14:51:29.073 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T14:51:29.074 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T14:51:29.101 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:51:29.194 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:51:29.284 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:51:29.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:51:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:51:29.410 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:51:29.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:51:29.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:51:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:51:29.478 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T14:51:29.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T14:51:29.495 INFO:teuthology.task.internal:Duration was 1174.920755 seconds 2023-10-18T14:51:29.496 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T14:51:29.505 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T14:51:29.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:51:29.508 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:51:29.510 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:51:29.554 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:51:29.558 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:51:29.567 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:51:29.866 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T14:51:29.867 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2023-10-18T14:51:29.867 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:51:29.890 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2023-10-18T14:51:29.891 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:51:29.915 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2023-10-18T14:51:29.916 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:51:29.940 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T14:51:29.940 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:51:29.943 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:51:29.957 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:51:30.017 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T14:51:30.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:51:30.020 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:51:30.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:51:30.117 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T14:51:30.127 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T14:51:30.127 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:51:30.155 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:51:30.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:51:30.211 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T14:51:30.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:51:30.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:51:30.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:51:30.247 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2023-10-18T14:51:30.249 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2023-10-18T14:51:30.277 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2023-10-18T14:51:30.301 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:51:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:51:30.342 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:51:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:51:30.360 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:51:30.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:51:30.378 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T14:51:30.387 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T14:51:30.389 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546/remote/smithi081 2023-10-18T14:51:30.391 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:51:30.446 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546/remote/smithi099 2023-10-18T14:51:30.447 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:51:30.504 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546/remote/smithi123 2023-10-18T14:51:30.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:51:30.551 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T14:51:30.552 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:51:30.555 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:51:30.561 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:51:30.603 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T14:51:30.616 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T14:51:30.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T14:51:30.625 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T14:51:30.626 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:51:30.628 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:51:30.630 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:51:30.643 INFO:teuthology.orchestra.run.smithi081.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:51 /home/ubuntu/cephtest 2023-10-18T14:51:30.646 INFO:teuthology.orchestra.run.smithi099.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:51 /home/ubuntu/cephtest 2023-10-18T14:51:30.661 INFO:teuthology.orchestra.run.smithi123.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:51 /home/ubuntu/cephtest 2023-10-18T14:51:30.662 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T14:51:30.672 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T14:51:30.692 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T14:51:30.716 DEBUG:teuthology.nuke:shortname: smithi081 2023-10-18T14:51:30.717 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:51:30.738 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:23:25.637787', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBoQdW73cZxv1fom3GBWWVhUCxE9Oarq0oP42ywimqxZkh3NoCCHKiDLO2TUJXT4Zhyr3WS5SYEuPVHHgd/2Hcc='} 2023-10-18T14:51:30.743 DEBUG:teuthology.nuke:shortname: smithi099 2023-10-18T14:51:30.743 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:51:30.767 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:23:25.639188', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNjhzcZjIyDK+DJtLZSLYBmd53yrAiCerwyGLgyfsCkCtoh6jPicrhINtSaU8kBZTGmwjpKRzj9GSxDCgMGy4+Y='} 2023-10-18T14:51:30.771 DEBUG:teuthology.nuke:shortname: smithi123 2023-10-18T14:51:30.771 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:51:30.791 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431546', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:23:25.636107', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGTKe4JOv6NNM33khpORUsC0CVo2m+62lOt5hfn4XcolFnZHXCNKks3eOGVw1nRAB3oN913vi9Pz+8yg3rEXuNI='} 2023-10-18T14:51:30.802 INFO:teuthology.orchestra.console.smithi081:Power off 2023-10-18T14:51:30.803 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:51:30.829 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2023-10-18T14:51:30.830 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:51:30.846 INFO:teuthology.orchestra.console.smithi099:Power off 2023-10-18T14:51:30.847 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:51:30.860 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2023-10-18T14:51:30.871 DEBUG:teuthology.orchestra.console.smithi099:power off output: Chassis Power Control: Down/Off 2023-10-18T14:51:30.871 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:51:30.887 INFO:teuthology.orchestra.console.smithi123:Power off 2023-10-18T14:51:30.888 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:51:30.897 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2023-10-18T14:51:30.909 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2023-10-18T14:51:30.910 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:51:30.932 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2023-10-18T14:51:34.862 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:51:34.898 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:51:34.934 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:51:34.973 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2023-10-18T14:51:35.008 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2023-10-18T14:51:35.046 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2023-10-18T14:51:38.978 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:51:39.009 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:51:39.047 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:51:39.091 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2023-10-18T14:51:39.120 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is off 2023-10-18T14:51:39.160 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2023-10-18T14:51:39.192 INFO:teuthology.orchestra.console.smithi081:Power off completed 2023-10-18T14:51:39.221 INFO:teuthology.orchestra.console.smithi099:Power off completed 2023-10-18T14:51:39.262 INFO:teuthology.orchestra.console.smithi123:Power off completed 2023-10-18T14:51:39.368 INFO:teuthology.run:Summary data: description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_cli_import_export}} duration: 1174.9207546710968 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 504 - HTTP Error 504: Gateway Timeout'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5a9c0082ce154f02aa3f1bf155db5373 status: dead success: false 2023-10-18T14:51:39.369 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:51:39.451 INFO:teuthology.run:DEAD