2023-12-04T18:48:31.551 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T18:48:31.552 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} --verbose --name mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi --archive /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296 --owner scheduled_mkogan@teuthology -- /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296/orig.config.yaml 2023-12-04T18:48:31.593 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T18:48:31.679 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296 branch: rgw-wip-t63445-valg-supp-rb_i001 description: rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} email: mkogan@redhat.com first_in_suite: false job_id: '7477296' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: rgw-wip-t63445-valg-supp-rb_i001 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on osd down out interval zero: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max pg log entries: 2 osd min pg log entries: 1 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled - overall HEALTH_ - \(MON_DOWN\) - \(MGR_DOWN\) - \(OSD_DOWN\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - slow request - failed to encode map sha1: 8c6118eee0f4440cb3687f9c8e2b718fd2d59013 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8c6118eee0f4440cb3687f9c8e2b718fd2d59013 ragweed: default-branch: ceph-master rgw_server: client.0 rgw: frontend: beast selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-qa-rgw-verify-ubuntu-004 sha1: 68a031970822325a492fbc35c67e2cf565873067 owner: scheduled_mkogan@teuthology priority: 150 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 1006 sha1: 8c6118eee0f4440cb3687f9c8e2b718fd2d59013 sleep_before_teardown: 0 suite: rgw suite_branch: wip-qa-rgw-verify-ubuntu-004 suite_path: /home/teuthworker/src/github.com_mkogan1_ceph_68a031970822325a492fbc35c67e2cf565873067/qa suite_relpath: qa suite_repo: https://github.com/mkogan1/ceph.git suite_sha1: 68a031970822325a492fbc35c67e2cf565873067 targets: smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNqcCJ7dyi8v/TPe5WG3P+wduZjH/kR9QMs8E46gXC/ifJaIKAXm6mfdVam9KXYzwZMn7KVo0FqV2lIVMfOquK8= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJRfawGpgrNR/lJlD8Ig2HTX5G5yOskYmk4EWVhsbGKWOalFd8aaRMMBjFOP77z90mlcMJgeQFrHh7+xyHXAj8= tasks: - install: branch: pacific exclude_packages: - ceph-volume - ceph: null - rgw: - client.0 - client.1 - tox: - client.0 - client.1 - print: installing upgraded packages - install.upgrade: mon.a: null mon.b: null - print: ragweed prepare before upgrade - ragweed: client.0: stages: prepare - print: restarting upgraded rgws - ceph.restart: daemons: - rgw.* - print: ragweed check/prepare after rgw upgrade - ragweed: client.0: stages: check client.1: stages: prepare - print: restarting upgraded osds - ceph.restart: daemons: - osd.0 - osd.2 - ceph.restart: daemons: - osd.1 - osd.3 - ceph.restart: daemons: - osd.4 - osd.6 - ceph.restart: daemons: - osd.5 - osd.7 - print: ragweed check after osd upgrade - ragweed: client.1: stages: check teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_18:39:17 tube: smithi user: mkogan verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T18:48:31.679 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mkogan1_ceph_68a031970822325a492fbc35c67e2cf565873067/qa; will attempt to use it 2023-12-04T18:48:31.680 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mkogan1_ceph_68a031970822325a492fbc35c67e2cf565873067/qa/tasks 2023-12-04T18:48:31.680 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T18:48:31.681 INFO:teuthology.task.internal:Checking packages... 2023-12-04T18:48:31.704 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8c6118eee0f4440cb3687f9c8e2b718fd2d59013' 2023-12-04T18:48:31.704 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T18:48:31.705 INFO:teuthology.packaging:ref: None 2023-12-04T18:48:31.705 INFO:teuthology.packaging:tag: None 2023-12-04T18:48:31.705 INFO:teuthology.packaging:branch: rgw-wip-t63445-valg-supp-rb_i001 2023-12-04T18:48:31.705 INFO:teuthology.packaging:sha1: 8c6118eee0f4440cb3687f9c8e2b718fd2d59013 2023-12-04T18:48:31.705 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=rgw-wip-t63445-valg-supp-rb_i001 2023-12-04T18:48:31.924 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7617.g8c6118ee 2023-12-04T18:48:31.924 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T18:48:31.933 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T18:48:31.933 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T18:48:31.956 INFO:teuthology.task.internal:Saving configuration 2023-12-04T18:48:31.972 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T18:48:31.980 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T18:48:32.002 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 18:40:16.816041', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNqcCJ7dyi8v/TPe5WG3P+wduZjH/kR9QMs8E46gXC/ifJaIKAXm6mfdVam9KXYzwZMn7KVo0FqV2lIVMfOquK8='} 2023-12-04T18:48:32.026 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 18:40:16.814543', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJRfawGpgrNR/lJlD8Ig2HTX5G5yOskYmk4EWVhsbGKWOalFd8aaRMMBjFOP77z90mlcMJgeQFrHh7+xyHXAj8='} 2023-12-04T18:48:32.027 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T18:48:32.034 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-04T18:48:32.034 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-04T18:48:32.034 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T18:48:32.109 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4964f69a60>, signals=[15]) 2023-12-04T18:48:32.110 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T18:48:32.116 INFO:teuthology.task.internal:Opening connections... 2023-12-04T18:48:32.116 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2023-12-04T18:48:32.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:48:32.187 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2023-12-04T18:48:32.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:48:32.254 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T18:48:32.262 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2023-12-04T18:48:32.285 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2023-12-04T18:48:32.285 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2023-12-04T18:48:32.345 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T18:48:32.346 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T18:48:32.347 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2023-12-04T18:48:32.371 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2023-12-04T18:48:32.394 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-12-04T18:48:32.395 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2023-12-04T18:48:32.456 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2023-12-04T18:48:32.456 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8" 2023-12-04T18:48:32.456 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2023-12-04T18:48:32.457 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2023-12-04T18:48:32.457 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8" 2023-12-04T18:48:32.457 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2023-12-04T18:48:32.457 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T18:48:32.457 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-12-04T18:48:32.457 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T18:48:32.457 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2023-12-04T18:48:32.457 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T18:48:32.457 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T18:48:32.458 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T18:48:32.458 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-12-04T18:48:32.479 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T18:48:32.490 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T18:48:32.497 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T18:48:32.497 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2023-12-04T18:48:32.499 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2023-12-04T18:48:32.520 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T18:48:32.527 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T18:48:32.527 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2023-12-04T18:48:32.560 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2023-12-04T18:48:32.594 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T18:48:32.650 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T18:48:32.666 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T18:48:32.666 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T18:48:32.667 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T18:48:32.667 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T18:48:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:48:32.686 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-12-04T18:48:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:48:32.688 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-12-04T18:48:32.746 INFO:teuthology.orchestra.run.smithi150.stdout:4.18.0-489.el8.x86_64 2023-12-04T18:48:32.746 INFO:teuthology.task.kernel:Running kernel on smithi150: 4.18.0-489.el8.x86_64 2023-12-04T18:48:32.746 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2023-12-04T18:48:32.748 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-489.el8.x86_64 2023-12-04T18:48:32.748 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-489.el8.x86_64 2023-12-04T18:48:32.748 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2023-12-04T18:48:37.957 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-12-04T18:48:38.200 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-12-04T18:49:03.465 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 8 - BaseOS 2.9 MB/s | 55 MB 00:19 2023-12-04T18:49:04.829 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - BaseOS 2.7 MB/s | 55 MB 00:20 2023-12-04T18:57:23.067 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 8 - Extras 37 B/s | 18 kB 08:11 2023-12-04T18:57:23.613 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-12-04T18:57:23.936 INFO:teuthology.orchestra.run.smithi150.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-12-04T18:57:25.372 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-12-04T18:57:29.169 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 377 kB/s | 24 kB 00:00 2023-12-04T18:57:32.844 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T18:57:32.845 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T18:57:33.158 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-04T18:57:33.160 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-04T18:57:33.160 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-12-04T18:57:33.160 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-04T18:57:33.160 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-04T18:57:33.160 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T18:57:33.160 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-04T18:57:33.160 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T18:57:33.160 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T18:57:33.161 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-04T18:57:33.161 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-04T18:57:33.161 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-04T18:57:33.161 INFO:teuthology.orchestra.run.smithi150.stdout:Install 3 Packages 2023-12-04T18:57:33.161 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-04T18:57:33.161 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 90 M 2023-12-04T18:57:33.161 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 96 M 2023-12-04T18:57:33.161 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-04T18:57:35.288 INFO:teuthology.orchestra.run.smithi150.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 5.2 MB/s | 10 MB 00:01 2023-12-04T18:57:36.533 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras 36 B/s | 18 kB 08:23 2023-12-04T18:57:37.051 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2023-12-04T18:57:37.354 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-12-04T18:57:37.646 INFO:teuthology.orchestra.run.smithi150.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 8.4 MB/s | 36 MB 00:04 2023-12-04T18:57:40.004 INFO:teuthology.orchestra.run.smithi150.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 6.5 MB/s | 43 MB 00:06 2023-12-04T18:57:40.004 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-04T18:57:40.004 INFO:teuthology.orchestra.run.smithi150.stdout:Total 13 MB/s | 90 MB 00:06 2023-12-04T18:57:40.679 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-04T18:57:41.095 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-04T18:57:41.095 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-04T18:57:42.342 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-04T18:57:42.373 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-04T18:57:43.702 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 16 MB 00:06 2023-12-04T18:57:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-04T18:57:45.736 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:57:47.527 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 421 kB/s | 24 kB 00:00 2023-12-04T18:57:49.294 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:57:49.717 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:57:51.126 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T18:57:51.126 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T18:57:51.429 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2023-12-04T18:57:51.430 INFO:teuthology.orchestra.run.smithi148.stdout:Install 3 Packages 2023-12-04T18:57:51.431 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-04T18:57:51.431 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 90 M 2023-12-04T18:57:51.431 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 96 M 2023-12-04T18:57:51.431 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2023-12-04T18:57:53.156 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:57:53.447 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:57:55.020 INFO:teuthology.orchestra.run.smithi148.stdout:(1/3): kernel-core-4.18.0-526.el8.x86_64.rpm 13 MB/s | 43 MB 00:03 2023-12-04T18:57:55.836 INFO:teuthology.orchestra.run.smithi148.stdout:(2/3): kernel-4.18.0-526.el8.x86_64.rpm 2.4 MB/s | 10 MB 00:04 2023-12-04T18:58:00.087 INFO:teuthology.orchestra.run.smithi148.stdout:(3/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 4.3 MB/s | 36 MB 00:08 2023-12-04T18:58:00.087 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2023-12-04T18:58:00.087 INFO:teuthology.orchestra.run.smithi148.stdout:Total 10 MB/s | 90 MB 00:08 2023-12-04T18:58:00.764 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2023-12-04T18:58:01.191 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2023-12-04T18:58:01.191 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2023-12-04T18:58:02.419 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2023-12-04T18:58:02.429 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2023-12-04T18:58:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2023-12-04T18:58:05.709 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:58:09.293 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:58:09.683 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:58:13.113 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:58:13.399 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:58:15.391 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:58:15.416 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:58:15.804 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:58:15.805 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:58:15.805 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:58:16.585 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:58:16.585 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-04T18:58:16.585 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-04T18:58:16.585 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T18:58:16.585 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T18:58:16.585 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-04T18:58:16.585 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-04T18:58:17.349 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T18:58:17.564 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T18:58:17.564 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T18:58:17.565 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T18:58:17.565 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T18:58:17.565 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T18:58:17.565 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-04T18:58:17.565 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T18:58:17.565 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T18:58:17.776 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T18:58:17.777 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T18:58:17.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.02 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 129.el8 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Group : System Environment/Base 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 9265586 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T18:58:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:Relocations : (not relocatable) 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : CentOS BuildSystem 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-04T18:58:17.812 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T18:58:17.813 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T18:58:17.813 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 4.18.0-526.el8.x86_64 2023-12-04T18:58:17.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T18:58:19.282 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2023-12-04T18:58:21.869 INFO:teuthology.orchestra.run.smithi150.stderr:done 2023-12-04T18:58:21.870 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2023-12-04T18:58:21.903 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.SO4NbrrT2x 2023-12-04T18:58:21.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SO4NbrrT2x 2023-12-04T18:58:21.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.SO4NbrrT2x 2023-12-04T18:58:22.126 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.SO4NbrrT2x is 6KB 2023-12-04T18:58:22.182 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.SO4NbrrT2x 2023-12-04T18:58:22.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T18:58:22.268 INFO:teuthology.orchestra.run.smithi150.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T18:58:22.268 INFO:teuthology.orchestra.run.smithi150.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T18:58:22.268 INFO:teuthology.orchestra.run.smithi150.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T18:58:22.268 INFO:teuthology.orchestra.run.smithi150.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T18:58:22.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T18:58:22.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2023-12-04T18:58:35.166 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:58:35.190 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:58:35.526 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:58:35.526 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T18:58:35.526 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T18:58:36.291 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T18:58:36.292 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-04T18:58:36.292 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2023-12-04T18:58:36.292 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T18:58:36.292 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T18:58:36.292 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-04T18:58:36.292 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-12-04T18:58:36.428 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T18:58:36.649 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T18:58:36.649 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T18:58:36.649 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T18:58:36.649 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T18:58:36.649 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T18:58:36.650 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-04T18:58:36.650 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T18:58:36.650 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T18:58:36.861 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T18:58:36.861 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T18:58:36.862 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2023-12-04T18:58:36.896 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2023-12-04T18:58:36.896 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2023-12-04T18:58:36.896 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.02 2023-12-04T18:58:36.896 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 129.el8 2023-12-04T18:58:36.896 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2023-12-04T18:58:36.896 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Group : System Environment/Base 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 9265586 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Relocations : (not relocatable) 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : CentOS BuildSystem 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2023-12-04T18:58:36.897 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-04T18:58:36.898 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T18:58:36.898 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T18:58:36.898 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T18:58:36.898 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2023-12-04T18:58:36.898 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-12-04T18:58:36.898 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T18:58:36.899 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T18:58:36.899 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 4.18.0-526.el8.x86_64 2023-12-04T18:58:36.899 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T18:58:38.577 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2023-12-04T18:58:40.787 INFO:teuthology.orchestra.run.smithi148.stderr:done 2023-12-04T18:58:40.788 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2023-12-04T18:58:40.823 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.20XZ4pYuy0 2023-12-04T18:58:40.824 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.20XZ4pYuy0 2023-12-04T18:58:40.898 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.20XZ4pYuy0 2023-12-04T18:58:41.046 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.20XZ4pYuy0 is 6KB 2023-12-04T18:58:41.095 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.20XZ4pYuy0 2023-12-04T18:58:41.112 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T18:58:41.180 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T18:58:41.180 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T18:58:41.180 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T18:58:41.180 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T18:58:41.181 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T18:58:41.338 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2023-12-04T18:58:52.462 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-04T18:58:52.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-04T18:58:52.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:59:10.819 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-04T18:59:11.342 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-04T18:59:11.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-04T18:59:11.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:59:19.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-04T18:59:19.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:59:22.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-04T18:59:29.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-12-04T18:59:34.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-04T18:59:34.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:59:38.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-04T18:59:38.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:59:41.826 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-12-04T18:59:53.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-04T18:59:53.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T18:59:56.898 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-12-04T19:00:11.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-04T19:00:11.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T19:00:14.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-12-04T19:00:32.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-04T19:00:32.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T19:00:33.262 DEBUG:teuthology.orchestra.run.smithi148:> true 2023-12-04T19:00:33.563 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-12-04T19:00:33.564 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T19:00:33.564 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-12-04T19:00:33.588 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-526.el8.x86_64 2023-12-04T19:00:33.588 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T19:00:33.588 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T19:00:33.588 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T19:00:34.589 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-04T19:00:34.590 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T19:00:34.671 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2023-12-04T19:00:34.687 DEBUG:teuthology.parallel:result is None 2023-12-04T19:00:34.894 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T19:00:49.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-04T19:00:49.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T19:00:50.170 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-12-04T19:00:50.480 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-04T19:00:50.480 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-526.el8.x86_64"... 2023-12-04T19:00:50.480 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-12-04T19:00:50.501 INFO:teuthology.orchestra.run.smithi150.stdout:4.18.0-526.el8.x86_64 2023-12-04T19:00:50.501 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T19:00:50.501 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T19:00:50.501 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T19:00:51.503 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-04T19:00:51.503 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T19:00:51.572 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2023-12-04T19:00:51.594 DEBUG:teuthology.parallel:result is None 2023-12-04T19:00:51.594 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T19:00:51.604 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T19:00:51.605 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T19:00:51.607 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T19:00:51.628 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T19:00:51.635 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T19:00:51.642 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T19:00:51.642 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T19:00:51.670 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T19:00:51.713 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T19:00:51.721 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T19:00:51.721 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T19:00:51.751 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T19:00:51.789 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T19:00:51.798 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T19:00:51.806 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T19:00:51.817 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T19:00:51.820 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T19:00:51.828 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T19:00:51.828 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T19:00:51.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T19:00:51.874 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T19:00:51.884 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T19:00:51.885 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T19:00:51.913 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T19:00:51.933 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T19:00:52.021 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T19:00:52.069 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T19:00:52.158 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T19:00:52.197 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-12-04T19:00:52.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T19:00:52.263 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T19:00:52.313 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T19:00:52.372 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T19:00:52.421 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T19:00:52.464 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-04T19:00:52.464 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T19:00:52.532 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2023-12-04T19:00:52.536 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2023-12-04T19:00:52.588 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T19:00:52.640 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T19:00:53.144 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T19:00:53.153 INFO:teuthology.task.internal:Starting timer... 2023-12-04T19:00:53.153 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T19:00:53.228 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T19:00:53.239 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-04T19:00:53.239 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2023-12-04T19:00:53.330 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2023-12-04T19:00:53.331 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2023-12-04T19:00:53.392 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2023-12-04T19:00:53.394 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T19:00:53.394 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2023-12-04T19:00:53.427 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2023-12-04T19:00:53.428 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2023-12-04T19:00:53.466 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2023-12-04T19:00:53.466 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi148.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2023-12-04T19:00:53.466 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T19:00:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:00:53.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T19:00:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:00:53.535 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T19:00:53.535 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2023-12-04T19:00:53.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2023-12-04T19:00:53.603 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T19:00:53.612 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-04T19:00:53.663 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-04T19:00:54.113 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T19:00:54.206 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T19:00:54.207 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi148.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T19:16:13.083 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296/ansible_failures.yaml 2023-12-04T19:16:13.085 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T19:16:13.158 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b40ec7b8049d4aeb9732394a7939ccec Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T19:16:13.160 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T19:16:13.167 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T19:16:13.167 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T19:16:13.181 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T19:16:13.273 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T19:16:13.366 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T19:16:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:16:13.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T19:16:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:16:13.432 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T19:16:13.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T19:16:13.454 INFO:teuthology.task.internal:Duration was 920.300625 seconds 2023-12-04T19:16:13.454 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T19:16:13.469 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T19:16:13.469 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T19:16:13.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T19:16:13.518 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T19:16:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T19:16:13.791 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T19:16:13.792 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2023-12-04T19:16:13.792 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T19:16:13.815 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2023-12-04T19:16:13.815 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T19:16:13.838 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T19:16:13.838 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T19:16:13.858 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T19:16:13.913 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T19:16:13.914 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T19:16:13.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T19:16:14.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T19:16:14.023 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T19:16:14.023 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T19:16:14.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T19:16:14.076 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T19:16:14.088 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T19:16:14.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T19:16:14.116 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2023-12-04T19:16:14.143 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2023-12-04T19:16:14.188 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T19:16:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:16:14.205 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T19:16:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T19:16:14.246 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T19:16:14.257 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T19:16:14.258 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296/remote/smithi148 2023-12-04T19:16:14.259 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T19:16:14.308 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296/remote/smithi150 2023-12-04T19:16:14.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T19:16:14.351 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T19:16:14.351 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T19:16:14.353 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T19:16:14.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T19:16:14.416 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T19:16:14.416 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T19:16:14.429 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T19:16:14.429 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T19:16:14.431 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T19:16:14.446 INFO:teuthology.orchestra.run.smithi148.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 19:16 /home/ubuntu/cephtest 2023-12-04T19:16:14.461 INFO:teuthology.orchestra.run.smithi150.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 19:16 /home/ubuntu/cephtest 2023-12-04T19:16:14.462 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T19:16:14.473 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T19:16:14.497 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T19:16:14.525 DEBUG:teuthology.nuke:shortname: smithi148 2023-12-04T19:16:14.526 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T19:16:14.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 18:40:16.816041', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNqcCJ7dyi8v/TPe5WG3P+wduZjH/kR9QMs8E46gXC/ifJaIKAXm6mfdVam9KXYzwZMn7KVo0FqV2lIVMfOquK8='} 2023-12-04T19:16:14.554 DEBUG:teuthology.nuke:shortname: smithi150 2023-12-04T19:16:14.554 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T19:16:14.576 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-12-04_18:39:17-rgw-rgw-wip-t63445-valg-supp-rb_i001-distro-default-smithi/7477296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 18:40:16.814543', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJRfawGpgrNR/lJlD8Ig2HTX5G5yOskYmk4EWVhsbGKWOalFd8aaRMMBjFOP77z90mlcMJgeQFrHh7+xyHXAj8='} 2023-12-04T19:16:14.603 INFO:teuthology.orchestra.console.smithi148:Power off 2023-12-04T19:16:14.603 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T19:16:14.626 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2023-12-04T19:16:14.626 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:16:14.642 INFO:teuthology.orchestra.console.smithi150:Power off 2023-12-04T19:16:14.642 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T19:16:14.652 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-12-04T19:16:14.665 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2023-12-04T19:16:14.665 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:16:14.687 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-12-04T19:16:18.654 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:16:18.688 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:16:18.764 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-12-04T19:16:18.799 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-12-04T19:16:22.766 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:16:22.800 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:16:22.876 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-12-04T19:16:22.910 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-12-04T19:16:22.977 INFO:teuthology.orchestra.console.smithi148:Power off completed 2023-12-04T19:16:26.914 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T19:16:27.023 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2023-12-04T19:16:27.124 INFO:teuthology.orchestra.console.smithi150:Power off completed 2023-12-04T19:16:28.945 INFO:teuthology.run:Summary data: description: rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} duration: 920.3006250858307 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_mkogan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b40ec7b8049d4aeb9732394a7939ccec status: dead success: false 2023-12-04T19:16:28.946 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T19:16:29.030 INFO:teuthology.run:DEAD