2023-10-18T14:29:53.576 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-18T14:29:53.577 DEBUG:teuthology.run:Teuthology command: teuthology --description smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_python_api_tests}} --verbose --name yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547/orig.config.yaml 2023-10-18T14:29:53.612 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:29:53.697 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547 branch: quincy-release description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_python_api_tests}} email: null first_in_suite: false job_id: '7431547' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 10 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: quincy-release ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-quincy-test-fixes sha1: 69fb3d4792adb6f1b63824c294e62c636f9a850d owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mds.a - mgr.x - osd.0 - osd.1 - - mon.b - mds.b - mon.c - mgr.y - osd.2 - osd.3 - - client.0 seed: 7103 sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 sleep_before_teardown: 0 subset: 111/120000 suite: smoke suite_branch: wip-quincy-test-fixes suite_path: /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa suite_relpath: qa suite_repo: https://github.com/pdvian/ceph.git suite_sha1: 69fb3d4792adb6f1b63824c294e62c636f9a850d targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHotEKraf4N3BKbH026chPVLLU88Kz/fx7jrjvxp7P2xdONfBVQhrTSlfxs7pkfFlntbygvFPgLLd0IwM22HF4= smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZoVJp+lNgyVDrIY4b6WOf+sLInaCOHPLHqlqJaEyikfW1yjlLcljfvpPHTwrL/nrQ/JLC+Y3x7cCbnIsTlSIE= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGJQBLEPtRLLWJOAz2d/gxXhB1G/uv5AoLiDiuhDQRsVfFjqa/HbPLVD3dvI6yCznXakfOBYwnU83/irK+Zsrlw= tasks: - install: cleanup: true - ceph: log-ignorelist: - \(POOL_APP_NOT_ENABLED\) - ceph-fuse: null - workunit: clients: client.0: - rbd/test_librbd_python.sh env: RBD_FEATURES: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-18_13:55:44 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T14:29:53.697 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa; will attempt to use it 2023-10-18T14:29:53.698 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa/tasks 2023-10-18T14:29:53.699 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T14:29:53.700 INFO:teuthology.task.internal:Checking packages... 2023-10-18T14:29:53.724 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '9d6d32bb3452d5179bde2ee1cfa05df7a65f4586' 2023-10-18T14:29:53.724 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T14:29:53.725 INFO:teuthology.packaging:ref: None 2023-10-18T14:29:53.725 INFO:teuthology.packaging:tag: None 2023-10-18T14:29:53.725 INFO:teuthology.packaging:branch: quincy-release 2023-10-18T14:29:53.726 INFO:teuthology.packaging:sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 2023-10-18T14:29:53.726 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy-release 2023-10-18T14:29:54.094 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1415.g9d6d32bb 2023-10-18T14:29:54.094 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T14:29:54.101 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T14:29:54.101 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T14:29:54.169 INFO:teuthology.task.internal:Saving configuration 2023-10-18T14:29:54.182 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T14:29:54.188 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:29:54.214 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:24:06.125390', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHotEKraf4N3BKbH026chPVLLU88Kz/fx7jrjvxp7P2xdONfBVQhrTSlfxs7pkfFlntbygvFPgLLd0IwM22HF4='} 2023-10-18T14:29:54.239 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:24:06.128057', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZoVJp+lNgyVDrIY4b6WOf+sLInaCOHPLHqlqJaEyikfW1yjlLcljfvpPHTwrL/nrQ/JLC+Y3x7cCbnIsTlSIE='} 2023-10-18T14:29:54.264 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:24:06.126858', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGJQBLEPtRLLWJOAz2d/gxXhB1G/uv5AoLiDiuhDQRsVfFjqa/HbPLVD3dvI6yCznXakfOBYwnU83/irK+Zsrlw='} 2023-10-18T14:29:54.264 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T14:29:54.271 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-10-18T14:29:54.272 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-10-18T14:29:54.272 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['client.0'] 2023-10-18T14:29:54.272 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T14:29:54.388 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa5ec242a60>, signals=[15]) 2023-10-18T14:29:54.389 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T14:29:54.397 INFO:teuthology.task.internal:Opening connections... 2023-10-18T14:29:54.397 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T14:29:54.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:29:54.469 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2023-10-18T14:29:54.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:29:54.535 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2023-10-18T14:29:54.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:29:54.603 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T14:29:54.610 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2023-10-18T14:29:54.631 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2023-10-18T14:29:54.631 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2023-10-18T14:29:54.688 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:29:54.689 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T14:29:54.689 INFO:teuthology.orchestra.run.smithi003.stdout:ID="rhel" 2023-10-18T14:29:54.689 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="fedora" 2023-10-18T14:29:54.689 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8.4" 2023-10-18T14:29:54.690 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:29:54.690 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T14:29:54.690 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2023-10-18T14:29:54.690 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T14:29:54.690 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:29:54.691 INFO:teuthology.orchestra.run.smithi003.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:29:54.691 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:29:54.691 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T14:29:54.691 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:29:54.691 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T14:29:54.692 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:29:54.692 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T14:29:54.692 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2023-10-18T14:29:54.717 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2023-10-18T14:29:54.736 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2023-10-18T14:29:54.736 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2023-10-18T14:29:54.793 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:29:54.793 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T14:29:54.793 INFO:teuthology.orchestra.run.smithi046.stdout:ID="rhel" 2023-10-18T14:29:54.794 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="fedora" 2023-10-18T14:29:54.794 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="8.4" 2023-10-18T14:29:54.794 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:29:54.794 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T14:29:54.794 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2023-10-18T14:29:54.795 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T14:29:54.795 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:29:54.795 INFO:teuthology.orchestra.run.smithi046.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:29:54.795 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:29:54.795 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-18T14:29:54.795 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:29:54.796 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T14:29:54.796 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:29:54.796 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T14:29:54.796 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2023-10-18T14:29:54.825 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2023-10-18T14:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2023-10-18T14:29:54.842 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2023-10-18T14:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T14:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout:ID="rhel" 2023-10-18T14:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="fedora" 2023-10-18T14:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8.4" 2023-10-18T14:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T14:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2023-10-18T14:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T14:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-10-18T14:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T14:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T14:29:54.901 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2023-10-18T14:29:54.925 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T14:29:54.934 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T14:29:54.939 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T14:29:54.939 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:29:54.942 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:29:54.944 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:29:54.959 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T14:29:54.967 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T14:29:54.967 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:29:54.998 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:29:55.001 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:29:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T14:29:55.015 INFO:teuthology.orchestra.run.smithi046.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T14:29:55.016 INFO:teuthology.orchestra.run.smithi163.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T14:29:55.017 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T14:29:55.096 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T14:29:55.110 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-18T14:29:55.111 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-18T14:29:55.111 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:29:55.112 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:29:55.113 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:29:55.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:29:55.129 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2023-10-18T14:29:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:29:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:29:55.131 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-10-18T14:29:55.131 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-10-18T14:29:55.184 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:29:55.185 INFO:teuthology.task.kernel:Running kernel on smithi163: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:29:55.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2023-10-18T14:29:55.188 INFO:teuthology.orchestra.run.smithi046.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:29:55.188 INFO:teuthology.task.kernel:Running kernel on smithi046: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:29:55.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2023-10-18T14:29:55.189 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:29:55.190 INFO:teuthology.task.kernel:Running kernel on smithi003: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:29:55.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2023-10-18T14:29:56.680 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2023-10-18T14:29:56.681 INFO:teuthology.orchestra.run.smithi163.stdout:Unable to read consumer identity 2023-10-18T14:29:56.707 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-10-18T14:29:56.708 INFO:teuthology.orchestra.run.smithi046.stdout:Unable to read consumer identity 2023-10-18T14:29:56.746 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-10-18T14:29:56.746 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2023-10-18T14:29:58.077 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-10-18T14:29:58.110 INFO:teuthology.orchestra.run.smithi046.stdout:Copr repo for python3-asyncssh owned by ceph 32 kB/s | 3.5 kB 00:00 2023-10-18T14:29:58.171 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-10-18T14:29:59.241 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-18T14:29:59.301 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-18T14:29:59.741 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-10-18T14:30:02.593 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-10-18T14:30:02.734 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2023-10-18T14:30:03.205 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2023-10-18T14:30:03.404 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T14:30:03.404 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T14:30:03.405 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T14:30:03.538 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T14:30:03.538 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T14:30:03.539 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T14:30:03.572 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-10-18T14:30:03.573 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-10-18T14:30:03.574 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-10-18T14:30:03.689 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2023-10-18T14:30:03.691 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2023-10-18T14:30:03.691 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2023-10-18T14:30:03.699 DEBUG:teuthology.orchestra.run.smithi163:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:30:03.798 DEBUG:teuthology.orchestra.run.smithi046:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:30:03.987 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T14:30:03.988 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T14:30:03.988 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T14:30:04.047 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2023-10-18T14:30:04.047 INFO:teuthology.orchestra.run.smithi163.stdout:Unable to read consumer identity 2023-10-18T14:30:04.133 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T14:30:04.134 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T14:30:04.134 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T14:30:04.165 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-10-18T14:30:04.165 INFO:teuthology.orchestra.run.smithi046.stdout:Unable to read consumer identity 2023-10-18T14:30:04.251 DEBUG:teuthology.orchestra.run.smithi003:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:30:02 PM UTC. 2023-10-18T14:30:04.477 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:30:02 PM UTC. 2023-10-18T14:30:04.491 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:30:04.491 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:04.491 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:04.532 INFO:teuthology.orchestra.run.smithi163.stderr:Error: No packages marked for reinstall. 2023-10-18T14:30:04.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum reinstall -y kernel || true 2023-10-18T14:30:04.620 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-10-18T14:30:04.620 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2023-10-18T14:30:04.622 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:30:04.622 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:04.623 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:Error: No packages marked for reinstall. 2023-10-18T14:30:04.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum reinstall -y kernel || true 2023-10-18T14:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2023-10-18T14:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout:Unable to read consumer identity 2023-10-18T14:30:04.935 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:01 ago on Wed 18 Oct 2023 02:30:03 PM UTC. 2023-10-18T14:30:05.064 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-10-18T14:30:05.065 INFO:teuthology.orchestra.run.smithi046.stdout:Unable to read consumer identity 2023-10-18T14:30:05.087 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:30:05.087 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:05.088 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:05.109 INFO:teuthology.orchestra.run.smithi003.stderr:Error: No packages marked for reinstall. 2023-10-18T14:30:05.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum reinstall -y kernel || true 2023-10-18T14:30:05.227 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:03 ago on Wed 18 Oct 2023 02:30:02 PM UTC. 2023-10-18T14:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:05.371 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:03 ago on Wed 18 Oct 2023 02:30:02 PM UTC. 2023-10-18T14:30:05.373 INFO:teuthology.orchestra.run.smithi163.stderr:Error: No packages marked for reinstall. 2023-10-18T14:30:05.429 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:30:05.513 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:30:05.514 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:05.514 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:05.525 INFO:teuthology.orchestra.run.smithi046.stderr:Error: No packages marked for reinstall. 2023-10-18T14:30:05.527 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-10-18T14:30:05.528 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2023-10-18T14:30:05.555 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.555 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.555 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.556 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:30:05.556 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:30:05.556 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-18T14:30:05.556 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:30:05.557 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:30:05.575 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:30:05.681 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.681 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2023-10-18T14:30:05.705 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.705 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.705 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.705 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:30:05.706 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:30:05.706 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-18T14:30:05.706 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:30:05.706 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:30:05.709 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2023-10-18T14:30:05.709 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2023-10-18T14:30:05.709 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.02 2023-10-18T14:30:05.709 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 99.el8_4.1 2023-10-18T14:30:05.710 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2023-10-18T14:30:05.710 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T14:30:05.710 INFO:teuthology.orchestra.run.smithi163.stdout:Group : System Environment/Base 2023-10-18T14:30:05.710 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 9264631 2023-10-18T14:30:05.710 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2023-10-18T14:30:05.711 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T14:30:05.711 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T14:30:05.711 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T14:30:05.711 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T14:30:05.711 INFO:teuthology.orchestra.run.smithi163.stdout:Relocations : (not relocatable) 2023-10-18T14:30:05.712 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : Red Hat, Inc. 2023-10-18T14:30:05.712 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : Red Hat, Inc. 2023-10-18T14:30:05.712 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:30:05.712 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2023-10-18T14:30:05.712 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2023-10-18T14:30:05.713 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-10-18T14:30:05.713 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:30:05.713 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:30:05.713 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:30:05.713 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2023-10-18T14:30:05.714 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-10-18T14:30:05.714 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:30:05.715 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:30:05.715 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.715 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:30:05.831 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:30:03 PM UTC. 2023-10-18T14:30:05.836 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.837 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2023-10-18T14:30:05.866 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2023-10-18T14:30:05.866 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2023-10-18T14:30:05.867 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.02 2023-10-18T14:30:05.867 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 99.el8_4.1 2023-10-18T14:30:05.867 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2023-10-18T14:30:05.867 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T14:30:05.867 INFO:teuthology.orchestra.run.smithi046.stdout:Group : System Environment/Base 2023-10-18T14:30:05.868 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 9264631 2023-10-18T14:30:05.868 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2023-10-18T14:30:05.868 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T14:30:05.868 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T14:30:05.869 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T14:30:05.869 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T14:30:05.869 INFO:teuthology.orchestra.run.smithi046.stdout:Relocations : (not relocatable) 2023-10-18T14:30:05.869 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : Red Hat, Inc. 2023-10-18T14:30:05.869 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : Red Hat, Inc. 2023-10-18T14:30:05.870 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:30:05.870 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2023-10-18T14:30:05.870 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2023-10-18T14:30:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-18T14:30:05.870 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:30:05.870 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:30:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:30:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2023-10-18T14:30:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-18T14:30:05.871 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:30:05.872 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:30:05.873 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:05.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:30:05.957 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:30:05.958 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:05.958 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:30:05.969 INFO:teuthology.orchestra.run.smithi003.stderr:Error: No packages marked for reinstall. 2023-10-18T14:30:06.026 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:30:06.157 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:06.158 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:06.158 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:06.158 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:30:06.158 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:30:06.159 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-18T14:30:06.159 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:30:06.159 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:30:06.290 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:06.290 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:06.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2023-10-18T14:30:06.319 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2023-10-18T14:30:06.319 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2023-10-18T14:30:06.319 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.02 2023-10-18T14:30:06.319 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 99.el8_4.1 2023-10-18T14:30:06.320 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2023-10-18T14:30:06.320 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T14:30:06.320 INFO:teuthology.orchestra.run.smithi003.stdout:Group : System Environment/Base 2023-10-18T14:30:06.320 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 9264631 2023-10-18T14:30:06.320 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2023-10-18T14:30:06.321 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T14:30:06.321 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T14:30:06.321 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T14:30:06.321 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T14:30:06.322 INFO:teuthology.orchestra.run.smithi003.stdout:Relocations : (not relocatable) 2023-10-18T14:30:06.322 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : Red Hat, Inc. 2023-10-18T14:30:06.322 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : Red Hat, Inc. 2023-10-18T14:30:06.322 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:30:06.322 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2023-10-18T14:30:06.323 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2023-10-18T14:30:06.323 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T14:30:06.323 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:30:06.323 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:30:06.323 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:30:06.323 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2023-10-18T14:30:06.324 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T14:30:06.324 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:30:06.325 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:30:06.325 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:06.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:30:07.045 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2023-10-18T14:30:07.250 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2023-10-18T14:30:07.692 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2023-10-18T14:30:09.104 INFO:teuthology.orchestra.run.smithi163.stderr:done 2023-10-18T14:30:09.107 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2023-10-18T14:30:09.122 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.ggTUd8iRd8 2023-10-18T14:30:09.122 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ggTUd8iRd8 2023-10-18T14:30:09.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.ggTUd8iRd8 2023-10-18T14:30:09.356 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.ggTUd8iRd8 is 6KB 2023-10-18T14:30:09.405 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.ggTUd8iRd8 2023-10-18T14:30:09.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:30:09.488 INFO:teuthology.orchestra.run.smithi163.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T14:30:09.489 INFO:teuthology.orchestra.run.smithi163.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T14:30:09.489 INFO:teuthology.orchestra.run.smithi163.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T14:30:09.489 INFO:teuthology.orchestra.run.smithi163.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T14:30:09.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:done 2023-10-18T14:30:09.525 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2023-10-18T14:30:09.547 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.FdTgdUm0c6 2023-10-18T14:30:09.548 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FdTgdUm0c6 2023-10-18T14:30:09.599 INFO:teuthology.orchestra.run.smithi003.stderr:done 2023-10-18T14:30:09.601 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2023-10-18T14:30:09.615 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.9qJdgGJHOe 2023-10-18T14:30:09.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9qJdgGJHOe 2023-10-18T14:30:09.617 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.FdTgdUm0c6 2023-10-18T14:30:09.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2023-10-18T14:30:09.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.9qJdgGJHOe 2023-10-18T14:30:09.771 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.FdTgdUm0c6 is 6KB 2023-10-18T14:30:09.819 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.FdTgdUm0c6 2023-10-18T14:30:09.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:30:09.840 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.9qJdgGJHOe is 6KB 2023-10-18T14:30:09.888 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.9qJdgGJHOe 2023-10-18T14:30:09.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:30:09.905 INFO:teuthology.orchestra.run.smithi046.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T14:30:09.905 INFO:teuthology.orchestra.run.smithi046.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T14:30:09.906 INFO:teuthology.orchestra.run.smithi046.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T14:30:09.906 INFO:teuthology.orchestra.run.smithi046.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T14:30:09.907 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:09.971 INFO:teuthology.orchestra.run.smithi003.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T14:30:09.971 INFO:teuthology.orchestra.run.smithi003.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T14:30:09.972 INFO:teuthology.orchestra.run.smithi003.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T14:30:09.972 INFO:teuthology.orchestra.run.smithi003.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T14:30:09.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:30:10.050 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2023-10-18T14:30:10.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2023-10-18T14:30:39.709 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-18T14:30:39.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-10-18T14:30:39.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:30:40.053 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-18T14:30:40.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-18T14:30:40.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:30:40.178 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-18T14:30:40.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T14:30:40.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:30:58.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-10-18T14:30:58.426 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2023-10-18T14:30:58.682 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2023-10-18T14:31:07.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-10-18T14:31:07.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:07.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-18T14:31:07.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:07.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T14:31:07.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:10.234 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-10-18T14:31:10.491 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2023-10-18T14:31:22.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-10-18T14:31:22.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:22.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-18T14:31:22.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:25.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-10-18T14:31:40.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-10-18T14:31:40.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:40.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2023-10-18T14:31:55.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-18T14:31:55.933 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:31:56.415 DEBUG:teuthology.orchestra.run.smithi046:> true 2023-10-18T14:31:56.703 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-18T14:31:56.703 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T14:31:56.704 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-10-18T14:31:56.720 INFO:teuthology.orchestra.run.smithi046.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T14:31:56.721 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:31:56.721 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:31:56.721 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T14:31:57.723 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-18T14:31:57.723 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:31:57.837 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2023-10-18T14:31:57.852 DEBUG:teuthology.parallel:result is None 2023-10-18T14:31:58.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-10-18T14:32:07.687 DEBUG:teuthology.orchestra.remote:timed out 2023-10-18T14:32:16.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-10-18T14:32:16.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:32:18.139 DEBUG:teuthology.orchestra.run.smithi163:> true 2023-10-18T14:32:18.802 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-10-18T14:32:18.802 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T14:32:18.802 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2023-10-18T14:32:18.817 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T14:32:18.817 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:32:18.818 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:32:18.818 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T14:32:19.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T14:32:19.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:32:19.818 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2023-10-18T14:32:19.819 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:32:19.897 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2023-10-18T14:32:19.914 DEBUG:teuthology.parallel:result is None 2023-10-18T14:32:20.050 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-10-18T14:32:20.342 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T14:32:20.342 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T14:32:20.343 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-10-18T14:32:20.359 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T14:32:20.359 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:32:20.359 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:32:20.360 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T14:32:21.361 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-18T14:32:21.361 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:32:21.452 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2023-10-18T14:32:21.467 DEBUG:teuthology.parallel:result is None 2023-10-18T14:32:21.467 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T14:32:21.475 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T14:32:21.475 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:32:21.478 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:32:21.482 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:32:21.500 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T14:32:21.508 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T14:32:21.515 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T14:32:21.516 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:32:21.534 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:32:21.542 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:32:21.578 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T14:32:21.584 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T14:32:21.584 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:32:21.603 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:32:21.609 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:32:21.639 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:32:21.645 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:32:21.646 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:32:21.651 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:32:21.657 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:32:21.657 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:32:21.659 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T14:32:21.666 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T14:32:21.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:32:21.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:32:21.701 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:32:21.728 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T14:32:21.737 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T14:32:21.738 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:32:21.761 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:32:21.770 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:32:21.785 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:32:21.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:32:21.899 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:32:21.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:32:22.022 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T14:32:22.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:32:22.087 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:32:22.132 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:32:22.178 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:32:22.261 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:32:22.307 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-18T14:32:22.307 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:32:22.376 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:32:22.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:32:22.464 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:32:22.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:32:22.591 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-10-18T14:32:22.591 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:32:22.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2023-10-18T14:32:22.659 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2023-10-18T14:32:22.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2023-10-18T14:32:22.712 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:32:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:32:22.751 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:32:23.231 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T14:32:23.239 INFO:teuthology.task.internal:Starting timer... 2023-10-18T14:32:23.240 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T14:32:23.249 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T14:32:23.256 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T14:32:23.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2023-10-18T14:32:23.315 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: [ OK ] 2023-10-18T14:32:23.317 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2023-10-18T14:32:23.393 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: [ OK ] 2023-10-18T14:32:23.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2023-10-18T14:32:23.458 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: [ OK ] 2023-10-18T14:32:23.459 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T14:32:23.460 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2023-10-18T14:32:23.480 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2023-10-18T14:32:23.480 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2023-10-18T14:32:23.498 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2023-10-18T14:32:23.498 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2023-10-18T14:32:23.518 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2023-10-18T14:32:23.518 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2023-10-18T14:32:23.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:32:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:32:23.547 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:32:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:32:23.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:32:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:32:23.601 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T14:32:23.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:32:23.627 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:32:23.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:32:23.695 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T14:32:23.706 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-18T14:32:23.707 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T14:32:23.724 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T14:32:23.725 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi046.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T14:42:41.353 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547/ansible_failures.yaml 2023-10-18T14:42:41.355 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:42:41.431 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=974ac4ca7eaa4d1294807731f0414a0f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:42:41.433 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T14:42:41.443 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T14:42:41.443 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T14:42:41.456 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:42:41.550 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:42:41.637 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:42:41.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:42:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:42:41.756 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:42:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:42:41.787 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:42:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:42:41.816 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T14:42:41.825 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T14:42:41.838 INFO:teuthology.task.internal:Duration was 618.598550 seconds 2023-10-18T14:42:41.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T14:42:41.852 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T14:42:41.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:42:41.855 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:42:41.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:42:41.902 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:42:41.902 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:42:41.904 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:42:42.241 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T14:42:42.242 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T14:42:42.242 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:42:42.265 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2023-10-18T14:42:42.266 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:42:42.285 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2023-10-18T14:42:42.285 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:42:42.304 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T14:42:42.305 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:42:42.307 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:42:42.327 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:42:42.379 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T14:42:42.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:42:42.387 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:42:42.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:42:42.484 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T14:42:42.495 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T14:42:42.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:42:42.522 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:42:42.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:42:42.572 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T14:42:42.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:42:42.585 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:42:42.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:42:42.609 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2023-10-18T14:42:42.611 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2023-10-18T14:42:42.636 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2023-10-18T14:42:42.650 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:42:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:42:42.679 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:42:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:42:42.693 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:42:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:42:42.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T14:42:42.720 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T14:42:42.721 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547/remote/smithi003 2023-10-18T14:42:42.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:42:42.771 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547/remote/smithi046 2023-10-18T14:42:42.772 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:42:42.816 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547/remote/smithi163 2023-10-18T14:42:42.817 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:42:42.859 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T14:42:42.860 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:42:42.862 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:42:42.864 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:42:42.909 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T14:42:42.923 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T14:42:42.923 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T14:42:42.937 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T14:42:42.938 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:42:42.940 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:42:42.943 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:42:42.953 INFO:teuthology.orchestra.run.smithi003.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:42 /home/ubuntu/cephtest 2023-10-18T14:42:42.955 INFO:teuthology.orchestra.run.smithi046.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:42 /home/ubuntu/cephtest 2023-10-18T14:42:42.963 INFO:teuthology.orchestra.run.smithi163.stdout: 262176 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:42 /home/ubuntu/cephtest 2023-10-18T14:42:42.964 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T14:42:42.976 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T14:42:43.126 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T14:42:43.154 DEBUG:teuthology.nuke:shortname: smithi003 2023-10-18T14:42:43.155 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:42:43.179 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:24:06.125390', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHotEKraf4N3BKbH026chPVLLU88Kz/fx7jrjvxp7P2xdONfBVQhrTSlfxs7pkfFlntbygvFPgLLd0IwM22HF4='} 2023-10-18T14:42:43.183 DEBUG:teuthology.nuke:shortname: smithi046 2023-10-18T14:42:43.183 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:42:43.207 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:24:06.128057', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZoVJp+lNgyVDrIY4b6WOf+sLInaCOHPLHqlqJaEyikfW1yjlLcljfvpPHTwrL/nrQ/JLC+Y3x7cCbnIsTlSIE='} 2023-10-18T14:42:43.213 DEBUG:teuthology.nuke:shortname: smithi163 2023-10-18T14:42:43.213 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:42:43.237 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:24:06.126858', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGJQBLEPtRLLWJOAz2d/gxXhB1G/uv5AoLiDiuhDQRsVfFjqa/HbPLVD3dvI6yCznXakfOBYwnU83/irK+Zsrlw='} 2023-10-18T14:42:43.246 INFO:teuthology.orchestra.console.smithi003:Power off 2023-10-18T14:42:43.247 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:42:43.271 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2023-10-18T14:42:43.271 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:42:43.286 INFO:teuthology.orchestra.console.smithi046:Power off 2023-10-18T14:42:43.286 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:42:43.297 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-10-18T14:42:43.308 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2023-10-18T14:42:43.309 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:42:43.321 INFO:teuthology.orchestra.console.smithi163:Power off 2023-10-18T14:42:43.322 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:42:43.330 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2023-10-18T14:42:43.343 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2023-10-18T14:42:43.344 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:42:43.364 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2023-10-18T14:42:47.299 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:42:47.335 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:42:47.365 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:42:47.438 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-10-18T14:42:47.445 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2023-10-18T14:42:47.475 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2023-10-18T14:42:51.442 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:42:51.450 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:42:51.476 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:42:51.552 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2023-10-18T14:42:51.559 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2023-10-18T14:42:51.585 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2023-10-18T14:42:51.654 INFO:teuthology.orchestra.console.smithi003:Power off completed 2023-10-18T14:42:51.661 INFO:teuthology.orchestra.console.smithi046:Power off completed 2023-10-18T14:42:51.686 INFO:teuthology.orchestra.console.smithi163:Power off completed 2023-10-18T14:42:51.792 INFO:teuthology.run:Summary data: description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_python_api_tests}} duration: 618.5985503196716 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Connection timed out'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=974ac4ca7eaa4d1294807731f0414a0f status: dead success: false 2023-10-18T14:42:51.793 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:42:51.870 INFO:teuthology.run:DEAD