2023-10-18T14:26:05.669 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-18T14:26:05.671 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/libcephfs_interface_tests}} --name yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544 --verbose -- /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544/orig.config.yaml 2023-10-18T14:26:05.706 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:26:05.781 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544 branch: quincy-release description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/libcephfs_interface_tests}} email: null first_in_suite: false job_id: '7431544' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 10 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: quincy-release ceph: conf: client: debug client: 20 debug ms: 1 mds: debug mds: 20 debug ms: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-quincy-test-fixes sha1: 69fb3d4792adb6f1b63824c294e62c636f9a850d owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mds.a - mgr.x - osd.0 - osd.1 - - mon.b - mds.b - mon.c - mgr.y - osd.2 - osd.3 - - client.0 seed: 7103 sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 sleep_before_teardown: 0 subset: 111/120000 suite: smoke suite_branch: wip-quincy-test-fixes suite_path: /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa suite_relpath: qa suite_repo: https://github.com/pdvian/ceph.git suite_sha1: 69fb3d4792adb6f1b63824c294e62c636f9a850d targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJChBFndO44wJQI8L6ZRKRJUzGcB+vUPItdTwrWQKiBrTUDsPy6Hcr8HDgJUzEYS/t/dSS8FsRTFf1uWcA2d9FA= smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA67koxyJ4GrixbqOomjK/WetEACmJggZXZLD8v5hTWABSXMXiTp/M2MtNQOEMvtNF+Ps1DU+HA37cCxxAdVHWM= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8IzB3RWQzci5zXZZy5b9QfRBii7vLBCfcIkJfJhcp3QkBBLiSplMD0uCWGennqCFCBeUX9+WLf9lpr5NfExBY= tasks: - install: cleanup: true - ceph: log-ignorelist: - \(POOL_APP_NOT_ENABLED\) - ceph-fuse: null - workunit: clients: client.0: - libcephfs/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-18_13:55:44 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T14:26:05.781 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa; will attempt to use it 2023-10-18T14:26:05.782 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa/tasks 2023-10-18T14:26:05.782 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T14:26:05.784 INFO:teuthology.task.internal:Checking packages... 2023-10-18T14:26:05.807 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '9d6d32bb3452d5179bde2ee1cfa05df7a65f4586' 2023-10-18T14:26:05.807 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T14:26:05.808 INFO:teuthology.packaging:ref: None 2023-10-18T14:26:05.808 INFO:teuthology.packaging:tag: None 2023-10-18T14:26:05.808 INFO:teuthology.packaging:branch: quincy-release 2023-10-18T14:26:05.809 INFO:teuthology.packaging:sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 2023-10-18T14:26:05.809 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy-release 2023-10-18T14:26:06.087 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1415.g9d6d32bb 2023-10-18T14:26:06.087 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T14:26:06.094 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T14:26:06.094 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T14:26:06.131 INFO:teuthology.task.internal:Saving configuration 2023-10-18T14:26:06.145 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T14:26:06.155 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:26:06.180 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:24.466376', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJChBFndO44wJQI8L6ZRKRJUzGcB+vUPItdTwrWQKiBrTUDsPy6Hcr8HDgJUzEYS/t/dSS8FsRTFf1uWcA2d9FA='} 2023-10-18T14:26:06.204 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:24.465174', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA67koxyJ4GrixbqOomjK/WetEACmJggZXZLD8v5hTWABSXMXiTp/M2MtNQOEMvtNF+Ps1DU+HA37cCxxAdVHWM='} 2023-10-18T14:26:06.227 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:24.463341', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8IzB3RWQzci5zXZZy5b9QfRBii7vLBCfcIkJfJhcp3QkBBLiSplMD0uCWGennqCFCBeUX9+WLf9lpr5NfExBY='} 2023-10-18T14:26:06.227 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T14:26:06.234 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-10-18T14:26:06.235 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-10-18T14:26:06.235 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['client.0'] 2023-10-18T14:26:06.236 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T14:26:06.364 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f26c673ca60>, signals=[15]) 2023-10-18T14:26:06.365 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T14:26:06.371 INFO:teuthology.task.internal:Opening connections... 2023-10-18T14:26:06.372 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2023-10-18T14:26:06.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:06.444 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2023-10-18T14:26:06.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:06.516 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2023-10-18T14:26:06.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:06.587 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T14:26:06.593 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2023-10-18T14:26:06.612 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2023-10-18T14:26:06.612 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2023-10-18T14:26:06.669 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:26:06.669 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T14:26:06.670 INFO:teuthology.orchestra.run.smithi012.stdout:ID="rhel" 2023-10-18T14:26:06.670 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="fedora" 2023-10-18T14:26:06.670 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="8.4" 2023-10-18T14:26:06.670 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:26:06.670 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T14:26:06.671 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2023-10-18T14:26:06.671 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T14:26:06.671 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:26:06.671 INFO:teuthology.orchestra.run.smithi012.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:26:06.671 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:26:06.672 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-10-18T14:26:06.672 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:26:06.672 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T14:26:06.672 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:26:06.672 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T14:26:06.673 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2023-10-18T14:26:06.694 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2023-10-18T14:26:06.712 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2023-10-18T14:26:06.713 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2023-10-18T14:26:06.771 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:26:06.771 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T14:26:06.772 INFO:teuthology.orchestra.run.smithi072.stdout:ID="rhel" 2023-10-18T14:26:06.772 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="fedora" 2023-10-18T14:26:06.772 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8.4" 2023-10-18T14:26:06.773 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:26:06.773 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T14:26:06.773 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2023-10-18T14:26:06.773 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T14:26:06.774 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:26:06.774 INFO:teuthology.orchestra.run.smithi072.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:26:06.774 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:26:06.774 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-10-18T14:26:06.775 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:26:06.775 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T14:26:06.775 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:26:06.775 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T14:26:06.776 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2023-10-18T14:26:06.800 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2023-10-18T14:26:06.819 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2023-10-18T14:26:06.819 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2023-10-18T14:26:06.875 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:26:06.876 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T14:26:06.876 INFO:teuthology.orchestra.run.smithi167.stdout:ID="rhel" 2023-10-18T14:26:06.876 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="fedora" 2023-10-18T14:26:06.876 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="8.4" 2023-10-18T14:26:06.876 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:26:06.877 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T14:26:06.877 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2023-10-18T14:26:06.877 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T14:26:06.877 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:26:06.878 INFO:teuthology.orchestra.run.smithi167.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:26:06.878 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:26:06.878 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-18T14:26:06.878 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:26:06.878 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T14:26:06.879 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:26:06.879 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T14:26:06.879 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2023-10-18T14:26:06.902 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T14:26:06.911 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T14:26:06.927 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T14:26:06.927 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:26:06.930 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:26:06.933 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:26:06.948 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T14:26:06.954 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T14:26:06.954 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:26:06.986 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:26:06.989 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:26:07.001 INFO:teuthology.orchestra.run.smithi012.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T14:26:07.004 INFO:teuthology.orchestra.run.smithi072.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T14:26:07.005 INFO:teuthology.orchestra.run.smithi167.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T14:26:07.007 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T14:26:07.088 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T14:26:07.104 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-18T14:26:07.104 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-18T14:26:07.105 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:26:07.106 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:26:07.106 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:26:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:26:07.123 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2023-10-18T14:26:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:26:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:26:07.127 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-10-18T14:26:07.127 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2023-10-18T14:26:07.179 INFO:teuthology.orchestra.run.smithi167.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:07.179 INFO:teuthology.task.kernel:Running kernel on smithi167: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:07.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2023-10-18T14:26:07.183 INFO:teuthology.orchestra.run.smithi072.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:07.183 INFO:teuthology.task.kernel:Running kernel on smithi072: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:07.184 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2023-10-18T14:26:07.185 INFO:teuthology.orchestra.run.smithi012.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:07.185 INFO:teuthology.task.kernel:Running kernel on smithi012: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:07.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2023-10-18T14:26:08.566 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:08.567 INFO:teuthology.orchestra.run.smithi012.stdout:Unable to read consumer identity 2023-10-18T14:26:08.700 INFO:teuthology.orchestra.run.smithi167.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:08.701 INFO:teuthology.orchestra.run.smithi167.stdout:Unable to read consumer identity 2023-10-18T14:26:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:Unable to read consumer identity 2023-10-18T14:26:10.048 INFO:teuthology.orchestra.run.smithi012.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-10-18T14:26:10.058 INFO:teuthology.orchestra.run.smithi167.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2023-10-18T14:26:10.702 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2023-10-18T14:26:11.013 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-10-18T14:26:11.939 INFO:teuthology.orchestra.run.smithi072.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-10-18T14:26:12.662 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2023-10-18T14:26:14.165 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-10-18T14:26:14.298 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-10-18T14:26:14.906 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T14:26:14.906 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T14:26:14.906 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T14:26:15.055 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-10-18T14:26:15.057 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-10-18T14:26:15.057 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-10-18T14:26:15.099 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T14:26:15.099 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T14:26:15.099 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T14:26:15.172 DEBUG:teuthology.orchestra.run.smithi012:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:26:15.245 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2023-10-18T14:26:15.246 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2023-10-18T14:26:15.246 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2023-10-18T14:26:15.352 DEBUG:teuthology.orchestra.run.smithi167:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:26:15.541 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:15.542 INFO:teuthology.orchestra.run.smithi012.stdout:Unable to read consumer identity 2023-10-18T14:26:15.698 INFO:teuthology.orchestra.run.smithi167.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:15.698 INFO:teuthology.orchestra.run.smithi167.stdout:Unable to read consumer identity 2023-10-18T14:26:15.860 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:01 ago on Wed 18 Oct 2023 02:26:14 PM UTC. 2023-10-18T14:26:15.991 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:15.992 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:15.992 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:15.996 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:01 ago on Wed 18 Oct 2023 02:26:14 PM UTC. 2023-10-18T14:26:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:16.068 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2023-10-18T14:26:16.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum reinstall -y kernel || true 2023-10-18T14:26:16.139 INFO:teuthology.orchestra.run.smithi167.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:16.139 INFO:teuthology.orchestra.run.smithi167.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:16.140 INFO:teuthology.orchestra.run.smithi167.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:16.161 INFO:teuthology.orchestra.run.smithi167.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:16.211 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum reinstall -y kernel || true 2023-10-18T14:26:16.445 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:16.445 INFO:teuthology.orchestra.run.smithi012.stdout:Unable to read consumer identity 2023-10-18T14:26:16.556 INFO:teuthology.orchestra.run.smithi167.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:16.556 INFO:teuthology.orchestra.run.smithi167.stdout:Unable to read consumer identity 2023-10-18T14:26:16.750 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:26:14 PM UTC. 2023-10-18T14:26:16.845 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:26:14 PM UTC. 2023-10-18T14:26:16.846 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T14:26:16.846 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T14:26:16.846 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T14:26:16.884 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:16.884 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:16.884 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:16.949 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:16.971 INFO:teuthology.orchestra.run.smithi167.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:16.971 INFO:teuthology.orchestra.run.smithi167.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:16.971 INFO:teuthology.orchestra.run.smithi167.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:16.986 INFO:teuthology.orchestra.run.smithi167.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:17.010 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-10-18T14:26:17.011 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-10-18T14:26:17.011 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-10-18T14:26:17.037 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:17.079 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.079 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.080 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.080 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:26:17.080 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:26:17.080 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-18T14:26:17.081 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:26:17.081 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:17.134 DEBUG:teuthology.orchestra.run.smithi072:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:26:17.161 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.162 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.162 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.162 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:26:17.163 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:26:17.163 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-18T14:26:17.163 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:26:17.163 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.211 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.212 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2023-10-18T14:26:17.239 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2023-10-18T14:26:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2023-10-18T14:26:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.02 2023-10-18T14:26:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 99.el8_4.1 2023-10-18T14:26:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2023-10-18T14:26:17.240 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T14:26:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:Group : System Environment/Base 2023-10-18T14:26:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 9264631 2023-10-18T14:26:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2023-10-18T14:26:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T14:26:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T14:26:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T14:26:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T14:26:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:Relocations : (not relocatable) 2023-10-18T14:26:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : Red Hat, Inc. 2023-10-18T14:26:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : Red Hat, Inc. 2023-10-18T14:26:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:26:17.243 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2023-10-18T14:26:17.243 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2023-10-18T14:26:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-10-18T14:26:17.243 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:26:17.243 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:26:17.243 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:26:17.244 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2023-10-18T14:26:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-10-18T14:26:17.244 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:26:17.245 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:26:17.245 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.245 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:26:17.286 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.288 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.288 DEBUG:teuthology.orchestra.run.smithi167:> sudo rpm -qi grub2-tools 2023-10-18T14:26:17.315 INFO:teuthology.orchestra.run.smithi167.stdout:Name : grub2-tools 2023-10-18T14:26:17.315 INFO:teuthology.orchestra.run.smithi167.stdout:Epoch : 1 2023-10-18T14:26:17.316 INFO:teuthology.orchestra.run.smithi167.stdout:Version : 2.02 2023-10-18T14:26:17.316 INFO:teuthology.orchestra.run.smithi167.stdout:Release : 99.el8_4.1 2023-10-18T14:26:17.316 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: x86_64 2023-10-18T14:26:17.317 INFO:teuthology.orchestra.run.smithi167.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T14:26:17.317 INFO:teuthology.orchestra.run.smithi167.stdout:Group : System Environment/Base 2023-10-18T14:26:17.317 INFO:teuthology.orchestra.run.smithi167.stdout:Size : 9264631 2023-10-18T14:26:17.317 INFO:teuthology.orchestra.run.smithi167.stdout:License : GPLv3+ 2023-10-18T14:26:17.317 INFO:teuthology.orchestra.run.smithi167.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T14:26:17.317 INFO:teuthology.orchestra.run.smithi167.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T14:26:17.318 INFO:teuthology.orchestra.run.smithi167.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T14:26:17.318 INFO:teuthology.orchestra.run.smithi167.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T14:26:17.318 INFO:teuthology.orchestra.run.smithi167.stdout:Relocations : (not relocatable) 2023-10-18T14:26:17.319 INFO:teuthology.orchestra.run.smithi167.stdout:Packager : Red Hat, Inc. 2023-10-18T14:26:17.319 INFO:teuthology.orchestra.run.smithi167.stdout:Vendor : Red Hat, Inc. 2023-10-18T14:26:17.319 INFO:teuthology.orchestra.run.smithi167.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:26:17.319 INFO:teuthology.orchestra.run.smithi167.stdout:Summary : Support tools for GRUB. 2023-10-18T14:26:17.319 INFO:teuthology.orchestra.run.smithi167.stdout:Description : 2023-10-18T14:26:17.320 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-18T14:26:17.320 INFO:teuthology.orchestra.run.smithi167.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:26:17.320 INFO:teuthology.orchestra.run.smithi167.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:26:17.320 INFO:teuthology.orchestra.run.smithi167.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:26:17.321 INFO:teuthology.orchestra.run.smithi167.stdout:hardware devices. 2023-10-18T14:26:17.321 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-18T14:26:17.321 INFO:teuthology.orchestra.run.smithi167.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:26:17.322 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:26:17.322 INFO:teuthology.task.kernel:Updating grub on smithi167 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:26:17.478 INFO:teuthology.orchestra.run.smithi072.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:17.478 INFO:teuthology.orchestra.run.smithi072.stdout:Unable to read consumer identity 2023-10-18T14:26:17.774 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:00:01 ago on Wed 18 Oct 2023 02:26:16 PM UTC. 2023-10-18T14:26:17.912 INFO:teuthology.orchestra.run.smithi072.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:17.912 INFO:teuthology.orchestra.run.smithi072.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:17.912 INFO:teuthology.orchestra.run.smithi072.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:17.952 INFO:teuthology.orchestra.run.smithi072.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:18.002 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum reinstall -y kernel || true 2023-10-18T14:26:18.344 INFO:teuthology.orchestra.run.smithi072.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:18.345 INFO:teuthology.orchestra.run.smithi072.stdout:Unable to read consumer identity 2023-10-18T14:26:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2023-10-18T14:26:18.541 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2023-10-18T14:26:18.636 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:26:16 PM UTC. 2023-10-18T14:26:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:18.849 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:18.974 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:18.974 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:18.974 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:18.975 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:26:18.975 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:26:18.975 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-18T14:26:18.975 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:26:18.976 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:19.100 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:19.100 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:19.100 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2023-10-18T14:26:19.127 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2023-10-18T14:26:19.127 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2023-10-18T14:26:19.128 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.02 2023-10-18T14:26:19.128 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 99.el8_4.1 2023-10-18T14:26:19.128 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2023-10-18T14:26:19.128 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T14:26:19.128 INFO:teuthology.orchestra.run.smithi072.stdout:Group : System Environment/Base 2023-10-18T14:26:19.129 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 9264631 2023-10-18T14:26:19.129 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2023-10-18T14:26:19.129 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T14:26:19.129 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T14:26:19.129 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T14:26:19.130 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T14:26:19.130 INFO:teuthology.orchestra.run.smithi072.stdout:Relocations : (not relocatable) 2023-10-18T14:26:19.130 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : Red Hat, Inc. 2023-10-18T14:26:19.130 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : Red Hat, Inc. 2023-10-18T14:26:19.130 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:26:19.131 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2023-10-18T14:26:19.131 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2023-10-18T14:26:19.131 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-10-18T14:26:19.131 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:26:19.131 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:26:19.131 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:26:19.132 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2023-10-18T14:26:19.132 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-10-18T14:26:19.132 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:26:19.133 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:26:19.133 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:19.133 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:26:20.608 INFO:teuthology.orchestra.run.smithi012.stderr:done 2023-10-18T14:26:20.610 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-10-18T14:26:20.625 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.Y4qwpdojsG 2023-10-18T14:26:20.625 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Y4qwpdojsG 2023-10-18T14:26:20.647 INFO:teuthology.orchestra.run.smithi167.stderr:done 2023-10-18T14:26:20.649 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2023-10-18T14:26:20.663 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.o9tVcJ9S05 2023-10-18T14:26:20.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.o9tVcJ9S05 2023-10-18T14:26:20.693 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.Y4qwpdojsG 2023-10-18T14:26:20.732 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.o9tVcJ9S05 2023-10-18T14:26:20.852 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.Y4qwpdojsG is 6KB 2023-10-18T14:26:20.871 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2023-10-18T14:26:20.889 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.o9tVcJ9S05 is 6KB 2023-10-18T14:26:20.900 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.Y4qwpdojsG 2023-10-18T14:26:20.915 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:26:20.937 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.o9tVcJ9S05 2023-10-18T14:26:20.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:26:20.994 INFO:teuthology.orchestra.run.smithi012.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T14:26:20.995 INFO:teuthology.orchestra.run.smithi012.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T14:26:20.995 INFO:teuthology.orchestra.run.smithi012.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T14:26:20.995 INFO:teuthology.orchestra.run.smithi012.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T14:26:20.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:21.011 INFO:teuthology.orchestra.run.smithi167.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T14:26:21.012 INFO:teuthology.orchestra.run.smithi167.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T14:26:21.012 INFO:teuthology.orchestra.run.smithi167.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T14:26:21.012 INFO:teuthology.orchestra.run.smithi167.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T14:26:21.013 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:21.159 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2023-10-18T14:26:21.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2023-10-18T14:26:23.379 INFO:teuthology.orchestra.run.smithi072.stderr:done 2023-10-18T14:26:23.380 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2023-10-18T14:26:23.403 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.aiTiM1tJ0q 2023-10-18T14:26:23.403 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aiTiM1tJ0q 2023-10-18T14:26:23.473 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.aiTiM1tJ0q 2023-10-18T14:26:23.629 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.aiTiM1tJ0q is 6KB 2023-10-18T14:26:23.676 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.aiTiM1tJ0q 2023-10-18T14:26:23.690 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:26:23.752 INFO:teuthology.orchestra.run.smithi072.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T14:26:23.752 INFO:teuthology.orchestra.run.smithi072.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T14:26:23.753 INFO:teuthology.orchestra.run.smithi072.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T14:26:23.753 INFO:teuthology.orchestra.run.smithi072.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T14:26:23.754 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:23.915 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2023-10-18T14:26:51.186 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-18T14:26:51.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-10-18T14:26:51.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:51.207 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-18T14:26:51.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-18T14:26:51.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:53.919 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-18T14:26:53.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-10-18T14:26:53.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:09.562 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2023-10-18T14:27:09.563 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-10-18T14:27:12.378 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-10-18T14:27:18.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-10-18T14:27:18.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:18.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-18T14:27:18.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:21.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-10-18T14:27:21.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:21.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-10-18T14:27:24.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-10-18T14:27:33.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-18T14:27:33.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:36.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-10-18T14:27:36.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:36.954 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2023-10-18T14:27:48.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-10-18T14:27:48.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:52.026 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2023-10-18T14:27:55.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-10-18T14:28:07.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-10-18T14:28:07.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:28:07.296 DEBUG:teuthology.orchestra.run.smithi012:> true 2023-10-18T14:28:07.565 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-10-18T14:28:07.565 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T14:28:07.566 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-10-18T14:28:07.582 INFO:teuthology.orchestra.run.smithi012.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:07.582 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:07.582 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:28:07.582 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T14:28:08.584 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-18T14:28:08.584 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:28:08.667 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2023-10-18T14:28:08.684 DEBUG:teuthology.parallel:result is None 2023-10-18T14:28:10.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-10-18T14:28:10.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:28:11.197 DEBUG:teuthology.orchestra.run.smithi072:> true 2023-10-18T14:28:11.763 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-10-18T14:28:11.763 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T14:28:11.764 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2023-10-18T14:28:11.778 INFO:teuthology.orchestra.run.smithi072.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:11.778 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:11.779 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:28:11.779 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T14:28:12.780 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-18T14:28:12.780 DEBUG:teuthology.orchestra.run.smithi072:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:28:13.009 INFO:teuthology.orchestra.run.smithi072.stdout:ttyS1 2023-10-18T14:28:13.021 DEBUG:teuthology.parallel:result is None 2023-10-18T14:28:33.637 DEBUG:teuthology.orchestra.remote:timed out 2023-10-18T14:28:48.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-18T14:28:48.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:28:49.036 DEBUG:teuthology.orchestra.run.smithi167:> true 2023-10-18T14:28:49.242 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-18T14:28:49.243 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T14:28:49.243 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2023-10-18T14:28:49.297 INFO:teuthology.orchestra.run.smithi167.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:49.297 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:49.298 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:28:49.298 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T14:28:50.299 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2023-10-18T14:28:50.300 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:28:50.384 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2023-10-18T14:28:50.396 DEBUG:teuthology.parallel:result is None 2023-10-18T14:28:50.396 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T14:28:50.405 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T14:28:50.405 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:28:50.408 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:28:50.412 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:28:50.429 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T14:28:50.448 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T14:28:50.480 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T14:28:50.481 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:28:50.483 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:28:50.486 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:28:50.514 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T14:28:50.523 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T14:28:50.523 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:28:50.550 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:28:50.552 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:28:50.576 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:50.585 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:50.587 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:50.594 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:50.598 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:50.605 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:50.606 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T14:28:50.617 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T14:28:50.617 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:28:50.630 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:28:50.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:28:50.672 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T14:28:50.684 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T14:28:50.684 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:28:50.697 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:28:50.709 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:28:50.727 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:50.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:50.833 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:50.913 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:50.955 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-10-18T14:28:50.956 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:28:51.020 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:51.060 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:51.106 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:51.189 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:51.231 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-10-18T14:28:51.231 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:28:51.297 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:51.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:51.379 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:51.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:51.500 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-18T14:28:51.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:28:51.564 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2023-10-18T14:28:51.567 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2023-10-18T14:28:51.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2023-10-18T14:28:51.629 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:28:51.637 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:28:51.652 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:28:52.119 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T14:28:52.127 INFO:teuthology.task.internal:Starting timer... 2023-10-18T14:28:52.128 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T14:28:52.158 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T14:28:52.190 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T14:28:52.191 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2023-10-18T14:28:52.256 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: [ OK ] 2023-10-18T14:28:52.258 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2023-10-18T14:28:52.323 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: [ OK ] 2023-10-18T14:28:52.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo service auditd rotate 2023-10-18T14:28:52.384 INFO:teuthology.orchestra.run.smithi167.stdout:Rotating logs: [ OK ] 2023-10-18T14:28:52.386 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T14:28:52.386 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2023-10-18T14:28:52.411 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2023-10-18T14:28:52.412 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2023-10-18T14:28:52.465 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2023-10-18T14:28:52.466 DEBUG:teuthology.orchestra.run.smithi167:> /usr/sbin/getenforce 2023-10-18T14:28:52.492 INFO:teuthology.orchestra.run.smithi167.stdout:Permissive 2023-10-18T14:28:52.492 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi072.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi167.front.sepia.ceph.com': 'permissive'} 2023-10-18T14:28:52.493 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:28:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:28:52.522 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:28:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:28:52.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:28:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:28:52.578 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T14:28:52.578 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:28:52.604 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:28:52.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:28:52.655 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T14:28:52.667 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-18T14:28:52.668 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T14:28:52.683 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T14:28:52.684 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi072.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T14:38:57.880 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544/ansible_failures.yaml 2023-10-18T14:38:57.882 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:38:57.962 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b0a822d11c5b44ae9cb5a05f04d12fcf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:38:57.964 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T14:38:57.974 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T14:38:57.975 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T14:38:57.999 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:38:58.090 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:38:58.172 DEBUG:teuthology.orchestra.run.smithi167:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:38:58.261 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:38:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:58.294 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:38:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:58.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:38:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:58.354 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T14:38:58.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T14:38:58.410 INFO:teuthology.task.internal:Duration was 606.282463 seconds 2023-10-18T14:38:58.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T14:38:58.443 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T14:38:58.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:38:58.446 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:38:58.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:38:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:38:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:38:58.495 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:38:58.870 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T14:38:58.870 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2023-10-18T14:38:58.871 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:38:58.893 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2023-10-18T14:38:58.894 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:38:58.913 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2023-10-18T14:38:58.913 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:38:58.932 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T14:38:58.933 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:38:58.936 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:38:58.955 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:38:59.007 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T14:38:59.008 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:38:59.014 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:38:59.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:38:59.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T14:38:59.121 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T14:38:59.121 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:38:59.149 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:38:59.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:38:59.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T14:38:59.213 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:59.216 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:59.218 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:59.239 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2023-10-18T14:38:59.240 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2023-10-18T14:38:59.261 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2023-10-18T14:38:59.275 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:59.308 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:59.322 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:59.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T14:38:59.348 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T14:38:59.350 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544/remote/smithi012 2023-10-18T14:38:59.351 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:38:59.418 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544/remote/smithi072 2023-10-18T14:38:59.420 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:38:59.467 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544/remote/smithi167 2023-10-18T14:38:59.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:38:59.512 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T14:38:59.513 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:38:59.515 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:38:59.517 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:38:59.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T14:38:59.576 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T14:38:59.577 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T14:38:59.602 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T14:38:59.603 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:38:59.605 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:38:59.608 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:38:59.618 INFO:teuthology.orchestra.run.smithi012.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:38 /home/ubuntu/cephtest 2023-10-18T14:38:59.619 INFO:teuthology.orchestra.run.smithi072.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:38 /home/ubuntu/cephtest 2023-10-18T14:38:59.622 INFO:teuthology.orchestra.run.smithi167.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:38 /home/ubuntu/cephtest 2023-10-18T14:38:59.623 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T14:38:59.638 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T14:38:59.703 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T14:38:59.729 DEBUG:teuthology.nuke:shortname: smithi012 2023-10-18T14:38:59.729 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:38:59.757 DEBUG:teuthology.nuke:shortname: smithi072 2023-10-18T14:38:59.757 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:38:59.762 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:24.466376', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJChBFndO44wJQI8L6ZRKRJUzGcB+vUPItdTwrWQKiBrTUDsPy6Hcr8HDgJUzEYS/t/dSS8FsRTFf1uWcA2d9FA='} 2023-10-18T14:38:59.784 DEBUG:teuthology.nuke:shortname: smithi167 2023-10-18T14:38:59.784 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:38:59.790 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:24.465174', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA67koxyJ4GrixbqOomjK/WetEACmJggZXZLD8v5hTWABSXMXiTp/M2MtNQOEMvtNF+Ps1DU+HA37cCxxAdVHWM='} 2023-10-18T14:38:59.813 INFO:teuthology.orchestra.console.smithi012:Power off 2023-10-18T14:38:59.814 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:38:59.825 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:24.463341', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8IzB3RWQzci5zXZZy5b9QfRBii7vLBCfcIkJfJhcp3QkBBLiSplMD0uCWGennqCFCBeUX9+WLf9lpr5NfExBY='} 2023-10-18T14:38:59.839 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2023-10-18T14:38:59.839 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:38:59.862 INFO:teuthology.orchestra.console.smithi072:Power off 2023-10-18T14:38:59.863 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:38:59.872 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-10-18T14:38:59.885 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2023-10-18T14:38:59.886 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:38:59.900 INFO:teuthology.orchestra.console.smithi167:Power off 2023-10-18T14:38:59.901 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:38:59.911 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2023-10-18T14:38:59.924 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2023-10-18T14:38:59.925 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:38:59.951 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2023-10-18T14:39:03.874 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:03.912 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:03.953 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:03.984 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-10-18T14:39:04.021 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2023-10-18T14:39:04.062 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2023-10-18T14:39:07.986 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:08.022 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:08.063 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:08.096 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-10-18T14:39:08.132 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2023-10-18T14:39:08.174 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2023-10-18T14:39:12.099 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:12.133 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:12.175 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:12.208 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2023-10-18T14:39:12.243 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2023-10-18T14:39:12.286 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2023-10-18T14:39:12.310 INFO:teuthology.orchestra.console.smithi012:Power off completed 2023-10-18T14:39:12.344 INFO:teuthology.orchestra.console.smithi072:Power off completed 2023-10-18T14:39:12.388 INFO:teuthology.orchestra.console.smithi167:Power off completed 2023-10-18T14:39:12.493 INFO:teuthology.run:Summary data: description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/libcephfs_interface_tests}} duration: 606.2824630737305 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Connection timed out'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b0a822d11c5b44ae9cb5a05f04d12fcf status: dead success: false 2023-10-18T14:39:12.494 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:39:12.564 INFO:teuthology.run:DEAD