2023-12-04T18:50:48.556 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T18:50:48.557 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252 --owner scheduled_mkogan@teuthology --description rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} --name mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi -- /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252/orig.config.yaml 2023-12-04T18:50:48.605 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T18:50:48.689 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252 branch: rgw-wip-t63445-valg-supp-rb_i001 description: rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} email: mkogan@redhat.com first_in_suite: false job_id: '7477252' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: rgw-wip-t63445-valg-supp-rb_i001 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on osd down out interval zero: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max pg log entries: 2 osd min pg log entries: 1 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled - overall HEALTH_ - \(MON_DOWN\) - \(MGR_DOWN\) - \(OSD_DOWN\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - slow request - failed to encode map sha1: 8c6118eee0f4440cb3687f9c8e2b718fd2d59013 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8c6118eee0f4440cb3687f9c8e2b718fd2d59013 ragweed: default-branch: ceph-master rgw_server: client.0 rgw: frontend: beast selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-qa-rgw-verify-ubuntu-004 sha1: 68a031970822325a492fbc35c67e2cf565873067 owner: scheduled_mkogan@teuthology priority: 150 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 1006 sha1: 8c6118eee0f4440cb3687f9c8e2b718fd2d59013 sleep_before_teardown: 0 suite: rgw suite_branch: wip-qa-rgw-verify-ubuntu-004 suite_path: /home/teuthworker/src/github.com_mkogan1_ceph_68a031970822325a492fbc35c67e2cf565873067/qa suite_relpath: qa suite_repo: https://github.com/mkogan1/ceph.git suite_sha1: 68a031970822325a492fbc35c67e2cf565873067 targets: smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0XmrbQ1eMowwmZ4rUpbYk//f/TpYwb9Lomt3sHdZ8+NQBXd+azEURVsLZ0udXNfx63OnffDSJAU/I57KGmx4g= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLy1bmFuMLPz6UBBv268/ThrSAj4XlQV+zFmx1Tkoyn+k9lp94uNPHhb3Y2L2B5KjVXIh9h5wmfoYerUnwsctEY= tasks: - install: branch: pacific exclude_packages: - ceph-volume - ceph: null - rgw: - client.0 - client.1 - tox: - client.0 - client.1 - print: installing upgraded packages - install.upgrade: mon.a: null mon.b: null - print: ragweed prepare before upgrade - ragweed: client.0: stages: prepare - print: restarting upgraded osds - ceph.restart: daemons: - osd.0 - osd.2 - ceph.restart: daemons: - osd.1 - osd.3 - ceph.restart: daemons: - osd.4 - osd.6 - ceph.restart: daemons: - osd.5 - osd.7 - print: ragweed check/prepare after osd upgrade - ragweed: client.0: stages: check client.1: stages: prepare - print: restarting upgraded rgw - ceph.restart: daemons: - rgw.* - print: ragweed check after rgw upgrade - ragweed: client.1: stages: check teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_18:39:17 tube: smithi user: mkogan verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T18:50:48.689 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mkogan1_ceph_68a031970822325a492fbc35c67e2cf565873067/qa; will attempt to use it 2023-12-04T18:50:48.690 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mkogan1_ceph_68a031970822325a492fbc35c67e2cf565873067/qa/tasks 2023-12-04T18:50:48.690 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T18:50:48.691 INFO:teuthology.task.internal:Checking packages... 2023-12-04T18:50:48.715 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8c6118eee0f4440cb3687f9c8e2b718fd2d59013' 2023-12-04T18:50:48.715 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T18:50:48.715 INFO:teuthology.packaging:ref: None 2023-12-04T18:50:48.715 INFO:teuthology.packaging:tag: None 2023-12-04T18:50:48.715 INFO:teuthology.packaging:branch: rgw-wip-t63445-valg-supp-rb_i001 2023-12-04T18:50:48.715 INFO:teuthology.packaging:sha1: 8c6118eee0f4440cb3687f9c8e2b718fd2d59013 2023-12-04T18:50:48.715 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=rgw-wip-t63445-valg-supp-rb_i001 2023-12-04T18:50:48.856 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7617.g8c6118ee 2023-12-04T18:50:48.856 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T18:50:48.864 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T18:50:48.864 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T18:50:48.873 INFO:teuthology.task.internal:Saving configuration 2023-12-04T18:50:48.892 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T18:50:48.903 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T18:50:48.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 18:39:43.121492', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0XmrbQ1eMowwmZ4rUpbYk//f/TpYwb9Lomt3sHdZ8+NQBXd+azEURVsLZ0udXNfx63OnffDSJAU/I57KGmx4g='} 2023-12-04T18:50:48.956 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 18:39:43.119882', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLy1bmFuMLPz6UBBv268/ThrSAj4XlQV+zFmx1Tkoyn+k9lp94uNPHhb3Y2L2B5KjVXIh9h5wmfoYerUnwsctEY='} 2023-12-04T18:50:48.956 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T18:50:48.964 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-04T18:50:48.964 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-04T18:50:48.964 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T18:50:49.046 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f213b10da60>, signals=[15]) 2023-12-04T18:50:49.046 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T18:50:49.055 INFO:teuthology.task.internal:Opening connections... 2023-12-04T18:50:49.055 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2023-12-04T18:50:49.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:50:49.127 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2023-12-04T18:50:49.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:50:49.198 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T18:50:49.207 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2023-12-04T18:50:49.228 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2023-12-04T18:50:49.229 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2023-12-04T18:50:49.289 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="8" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="8" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el8" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T18:50:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T18:50:49.291 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2023-12-04T18:50:49.314 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2023-12-04T18:50:49.338 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2023-12-04T18:50:49.338 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2023-12-04T18:50:49.401 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T18:50:49.402 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T18:50:49.403 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-12-04T18:50:49.422 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T18:50:49.434 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T18:50:49.443 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T18:50:49.444 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2023-12-04T18:50:49.446 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2023-12-04T18:50:49.466 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T18:50:49.477 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T18:50:49.477 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2023-12-04T18:50:49.508 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2023-12-04T18:50:49.540 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T18:50:49.596 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T18:50:49.618 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T18:50:49.618 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T18:50:49.618 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T18:50:49.619 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T18:50:49.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:50:49.639 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2023-12-04T18:50:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:50:49.640 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2023-12-04T18:50:49.699 INFO:teuthology.orchestra.run.smithi192.stdout:4.18.0-489.el8.x86_64 2023-12-04T18:50:49.699 INFO:teuthology.task.kernel:Running kernel on smithi192: 4.18.0-489.el8.x86_64 2023-12-04T18:50:49.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2023-12-04T18:50:49.701 INFO:teuthology.orchestra.run.smithi082.stdout:4.18.0-489.el8.x86_64 2023-12-04T18:50:49.701 INFO:teuthology.task.kernel:Running kernel on smithi082: 4.18.0-489.el8.x86_64 2023-12-04T18:50:49.701 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2023-12-04T18:50:54.911 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - AppStream 13 MB/s | 34 MB 00:02 2023-12-04T18:50:55.167 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-12-04T18:51:07.975 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - BaseOS 8.3 MB/s | 55 MB 00:06 2023-12-04T18:51:12.626 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - BaseOS 4.8 MB/s | 55 MB 00:11 2023-12-04T18:51:16.133 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras 62 kB/s | 18 kB 00:00 2023-12-04T18:51:16.502 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras common packages 39 kB/s | 6.9 kB 00:00 2023-12-04T18:51:16.810 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-12-04T18:51:18.088 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-12-04T18:51:20.849 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2023-12-04T18:51:21.392 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-12-04T18:51:21.702 INFO:teuthology.orchestra.run.smithi082.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-12-04T18:51:22.069 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 165 kB/s | 24 kB 00:00 2023-12-04T18:51:23.129 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-12-04T18:51:25.702 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T18:51:25.702 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T18:51:25.999 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-12-04T18:51:25.999 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout:Install 3 Packages 2023-12-04T18:51:26.000 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-04T18:51:26.001 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 90 M 2023-12-04T18:51:26.001 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 96 M 2023-12-04T18:51:26.001 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-12-04T18:51:26.969 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 389 kB/s | 24 kB 00:00 2023-12-04T18:51:28.221 INFO:teuthology.orchestra.run.smithi192.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 4.6 MB/s | 10 MB 00:02 2023-12-04T18:51:30.659 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T18:51:30.659 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T18:51:30.965 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-12-04T18:51:30.966 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-12-04T18:51:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout:Install 3 Packages 2023-12-04T18:51:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T18:51:30.968 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 90 M 2023-12-04T18:51:30.968 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 96 M 2023-12-04T18:51:30.968 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-12-04T18:51:38.335 INFO:teuthology.orchestra.run.smithi082.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 1.4 MB/s | 10 MB 00:07 2023-12-04T18:51:38.653 INFO:teuthology.orchestra.run.smithi192.stdout:(2/3): kernel-core-4.18.0-526.el8.x86_64.rpm 3.4 MB/s | 43 MB 00:12 2023-12-04T18:51:44.902 INFO:teuthology.orchestra.run.smithi192.stdout:(3/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 1.9 MB/s | 36 MB 00:18 2023-12-04T18:51:44.903 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-12-04T18:51:44.903 INFO:teuthology.orchestra.run.smithi192.stdout:Total 4.7 MB/s | 90 MB 00:18 2023-12-04T18:51:45.574 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-12-04T18:51:46.036 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-12-04T18:51:46.036 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-12-04T18:51:47.285 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-12-04T18:51:47.319 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-12-04T18:51:47.675 INFO:teuthology.orchestra.run.smithi082.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.2 MB/s | 36 MB 00:16 2023-12-04T18:51:49.034 INFO:teuthology.orchestra.run.smithi082.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.4 MB/s | 43 MB 00:18 2023-12-04T18:51:49.034 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-12-04T18:51:49.034 INFO:teuthology.orchestra.run.smithi082.stdout:Total 5.0 MB/s | 90 MB 00:18 2023-12-04T18:51:49.715 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-12-04T18:51:50.116 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-12-04T18:51:50.144 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-12-04T18:51:50.144 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-12-04T18:51:50.644 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:51:51.391 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-12-04T18:51:51.434 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-12-04T18:51:54.210 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-12-04T18:51:54.281 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:51:54.688 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:51:54.739 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:51:58.121 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:51:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:51:58.382 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:51:58.922 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:52:02.345 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:52:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:52:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:52:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:52:20.532 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:52:20.532 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:52:20.532 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:52:21.259 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:52:21.259 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-04T18:52:21.259 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-12-04T18:52:21.260 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T18:52:21.260 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T18:52:21.260 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-04T18:52:21.260 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-12-04T18:52:21.404 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T18:52:21.624 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T18:52:21.625 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T18:52:21.625 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T18:52:21.625 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T18:52:21.625 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T18:52:21.625 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-04T18:52:21.625 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T18:52:21.625 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T18:52:21.834 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T18:52:21.834 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T18:52:21.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2023-12-04T18:52:21.870 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.02 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 129.el8 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Group : System Environment/Base 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 9265586 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T18:52:21.871 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:Relocations : (not relocatable) 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : CentOS BuildSystem 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-12-04T18:52:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T18:52:21.872 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T18:52:21.873 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 4.18.0-526.el8.x86_64 2023-12-04T18:52:21.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T18:52:23.530 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2023-12-04T18:52:24.550 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:52:24.576 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:52:25.206 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:52:25.206 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:52:25.206 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:52:25.811 INFO:teuthology.orchestra.run.smithi192.stderr:done 2023-12-04T18:52:25.812 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2023-12-04T18:52:25.850 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.DyZz2f1LfT 2023-12-04T18:52:25.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DyZz2f1LfT 2023-12-04T18:52:25.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.DyZz2f1LfT 2023-12-04T18:52:25.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:52:25.973 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T18:52:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-12-04T18:52:25.974 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T18:52:25.974 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T18:52:25.974 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T18:52:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-12-04T18:52:26.067 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.DyZz2f1LfT is 6KB 2023-12-04T18:52:26.116 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.DyZz2f1LfT 2023-12-04T18:52:26.123 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T18:52:26.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T18:52:26.208 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T18:52:26.208 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T18:52:26.208 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T18:52:26.209 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T18:52:26.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T18:52:26.342 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T18:52:26.342 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T18:52:26.342 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T18:52:26.343 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T18:52:26.343 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T18:52:26.343 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-04T18:52:26.343 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T18:52:26.343 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T18:52:26.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2023-12-04T18:52:26.552 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T18:52:26.552 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T18:52:26.552 DEBUG:teuthology.orchestra.run.smithi082:> sudo rpm -qi grub2-tools 2023-12-04T18:52:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:Name : grub2-tools 2023-12-04T18:52:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:Epoch : 1 2023-12-04T18:52:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:Version : 2.02 2023-12-04T18:52:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:Release : 129.el8 2023-12-04T18:52:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: x86_64 2023-12-04T18:52:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T18:52:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:Group : System Environment/Base 2023-12-04T18:52:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:Size : 9265586 2023-12-04T18:52:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:License : GPLv3+ 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:Relocations : (not relocatable) 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:Packager : CentOS BuildSystem 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:Vendor : CentOS 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:Summary : Support tools for GRUB. 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:Description : 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T18:52:26.591 INFO:teuthology.orchestra.run.smithi082.stdout:hardware devices. 2023-12-04T18:52:26.592 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T18:52:26.592 INFO:teuthology.orchestra.run.smithi082.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T18:52:26.592 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T18:52:26.592 INFO:teuthology.task.kernel:Updating grub on smithi082 to boot 4.18.0-526.el8.x86_64 2023-12-04T18:52:26.592 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T18:52:28.014 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2023-12-04T18:52:30.265 INFO:teuthology.orchestra.run.smithi082.stderr:done 2023-12-04T18:52:30.267 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2023-12-04T18:52:30.298 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.fm2ygIv2Wj 2023-12-04T18:52:30.299 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fm2ygIv2Wj 2023-12-04T18:52:30.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.fm2ygIv2Wj 2023-12-04T18:52:30.525 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.fm2ygIv2Wj is 6KB 2023-12-04T18:52:30.574 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.fm2ygIv2Wj 2023-12-04T18:52:30.593 DEBUG:teuthology.orchestra.run.smithi082:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T18:52:30.661 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T18:52:30.661 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T18:52:30.661 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T18:52:30.662 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T18:52:30.663 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T18:52:30.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2023-12-04T18:52:56.414 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-04T18:52:56.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-04T18:52:56.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:53:00.830 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-04T18:53:00.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-12-04T18:53:00.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:53:56.470 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T18:54:00.835 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T18:54:05.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-04T18:54:05.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:54:05.917 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-12-04T18:54:06.192 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-12-04T18:54:06.192 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-526.el8.x86_64"... 2023-12-04T18:54:06.192 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2023-12-04T18:54:06.253 INFO:teuthology.orchestra.run.smithi192.stdout:4.18.0-526.el8.x86_64 2023-12-04T18:54:06.253 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T18:54:06.253 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T18:54:06.253 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T18:54:07.255 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-04T18:54:07.255 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T18:54:07.353 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2023-12-04T18:54:07.371 DEBUG:teuthology.parallel:result is None 2023-12-04T18:54:09.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-12-04T18:54:09.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:54:10.258 DEBUG:teuthology.orchestra.run.smithi082:> true 2023-12-04T18:54:10.606 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-12-04T18:54:10.607 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T18:54:10.607 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2023-12-04T18:54:10.624 INFO:teuthology.orchestra.run.smithi082.stdout:4.18.0-526.el8.x86_64 2023-12-04T18:54:10.624 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T18:54:10.624 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T18:54:10.625 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T18:54:11.628 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-04T18:54:11.628 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T18:54:11.729 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2023-12-04T18:54:11.740 DEBUG:teuthology.parallel:result is None 2023-12-04T18:54:11.740 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T18:54:11.748 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T18:54:11.748 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T18:54:11.750 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T18:54:11.774 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T18:54:11.811 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T18:54:11.819 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T18:54:11.820 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T18:54:11.824 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T18:54:11.871 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T18:54:11.878 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T18:54:11.879 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T18:54:11.890 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T18:54:11.935 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T18:54:11.945 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T18:54:11.959 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T18:54:11.974 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T18:54:11.974 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T18:54:11.987 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T18:54:11.988 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T18:54:11.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T18:54:12.019 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T18:54:12.035 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T18:54:12.036 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T18:54:12.058 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T18:54:12.083 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T18:54:12.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T18:54:12.219 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T18:54:12.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T18:54:12.348 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-12-04T18:54:12.348 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T18:54:12.416 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T18:54:12.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T18:54:12.529 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T18:54:12.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T18:54:12.668 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-12-04T18:54:12.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T18:54:12.740 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2023-12-04T18:54:12.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2023-12-04T18:54:12.817 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T18:54:12.838 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T18:54:13.356 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T18:54:13.367 INFO:teuthology.task.internal:Starting timer... 2023-12-04T18:54:13.367 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T18:54:13.377 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T18:54:13.387 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-04T18:54:13.388 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2023-12-04T18:54:13.467 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2023-12-04T18:54:13.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2023-12-04T18:54:13.542 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2023-12-04T18:54:13.544 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T18:54:13.544 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2023-12-04T18:54:13.566 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2023-12-04T18:54:13.567 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2023-12-04T18:54:13.627 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2023-12-04T18:54:13.627 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi082.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2023-12-04T18:54:13.627 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T18:54:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:54:13.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T18:54:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:54:13.694 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T18:54:13.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2023-12-04T18:54:13.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2023-12-04T18:54:13.767 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T18:54:13.778 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-04T18:54:13.778 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T18:54:13.819 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T18:54:13.820 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi082.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T19:14:21.938 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252/ansible_failures.yaml 2023-12-04T19:14:23.061 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T19:14:23.138 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a2c4710881c943aab9eaeb4c4a015785 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T19:14:23.140 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T19:14:23.149 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T19:14:23.149 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T19:14:23.159 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T19:14:23.251 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T19:14:23.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T19:14:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:14:23.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T19:14:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:14:23.405 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T19:14:23.414 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T19:14:23.422 INFO:teuthology.task.internal:Duration was 1210.055021 seconds 2023-12-04T19:14:23.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T19:14:23.433 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T19:14:23.433 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T19:14:23.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T19:14:23.481 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T19:14:23.493 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T19:14:23.814 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T19:14:23.814 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2023-12-04T19:14:23.815 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T19:14:23.839 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2023-12-04T19:14:23.839 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T19:14:23.861 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T19:14:23.862 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T19:14:23.881 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T19:14:23.937 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T19:14:23.937 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T19:14:23.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T19:14:24.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T19:14:24.049 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T19:14:24.049 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T19:14:24.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T19:14:24.102 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T19:14:24.138 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T19:14:24.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T19:14:24.164 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2023-12-04T19:14:24.168 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2023-12-04T19:14:24.245 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T19:14:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:14:24.301 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T19:14:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:14:24.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T19:14:24.340 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T19:14:24.341 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252/remote/smithi082 2023-12-04T19:14:24.342 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T19:14:24.391 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252/remote/smithi192 2023-12-04T19:14:24.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T19:14:24.448 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T19:14:24.448 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T19:14:24.450 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T19:14:24.467 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T19:14:24.479 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T19:14:24.480 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T19:14:24.519 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T19:14:24.520 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T19:14:24.522 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T19:14:24.536 INFO:teuthology.orchestra.run.smithi082.stdout: 525551 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 19:14 /home/ubuntu/cephtest 2023-12-04T19:14:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 19:14 /home/ubuntu/cephtest 2023-12-04T19:14:24.540 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T19:14:24.557 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T19:14:24.584 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T19:14:24.610 DEBUG:teuthology.nuke:shortname: smithi082 2023-12-04T19:14:24.611 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T19:14:24.634 DEBUG:teuthology.nuke:shortname: smithi192 2023-12-04T19:14:24.634 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T19:14:24.638 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 18:39:43.121492', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0XmrbQ1eMowwmZ4rUpbYk//f/TpYwb9Lomt3sHdZ8+NQBXd+azEURVsLZ0udXNfx63OnffDSJAU/I57KGmx4g='} 2023-12-04T19:14:24.662 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 18:39:43.119882', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLy1bmFuMLPz6UBBv268/ThrSAj4XlQV+zFmx1Tkoyn+k9lp94uNPHhb3Y2L2B5KjVXIh9h5wmfoYerUnwsctEY='} 2023-12-04T19:14:24.696 INFO:teuthology.orchestra.console.smithi082:Power off 2023-12-04T19:14:24.697 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T19:14:24.712 INFO:teuthology.orchestra.console.smithi192:Power off 2023-12-04T19:14:24.712 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T19:14:24.722 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2023-12-04T19:14:24.723 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:14:24.734 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2023-12-04T19:14:24.735 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:14:24.747 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2023-12-04T19:14:24.761 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-12-04T19:14:28.749 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:14:28.762 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:14:28.859 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2023-12-04T19:14:28.871 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-12-04T19:14:32.864 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:14:32.873 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:14:32.976 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2023-12-04T19:14:32.983 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2023-12-04T19:14:33.084 INFO:teuthology.orchestra.console.smithi192:Power off completed 2023-12-04T19:14:36.981 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:14:37.091 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2023-12-04T19:14:37.192 INFO:teuthology.orchestra.console.smithi082:Power off completed 2023-12-04T19:14:37.297 INFO:teuthology.run:Summary data: description: rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} duration: 1210.0550208091736 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_mkogan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a2c4710881c943aab9eaeb4c4a015785 status: dead success: false 2023-12-04T19:14:37.297 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T19:14:37.404 INFO:teuthology.run:DEAD