2024-04-10T20:00:46.686 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T20:00:46.686 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130 --name yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} -- /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130/orig.config.yaml 2024-04-10T20:00:46.721 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T20:00:46.826 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130 branch: wip-yuri10-testing-2024-03-24-1159 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} email: yweinste@redhat.com first_in_suite: false job_id: '7651130' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-03-24-1159 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri10-testing-2024-03-24-1159 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 8148 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-03-24-1159 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHoJThUGdtFA9y8J50DIpJZrXDiHovtyIB+U9o9qf+Btdp8AvW353V0ElnF/bo01qbuR610FMF3B67WwggfI/6k= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2XvNrpWdbEZ+0vSPo/qWhsqOSDd4WxfCt2DKfs1Gj0f8TTUCMqpWVFtutnjdAGpH6wSLOvOhHXazf8SWOh0Sc= smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB+woxYWjV74JfpnRSuShuFxv2xTDMaFxtFWjZgYky/hUGaHnRftBF8L39c8gbg4EsCkgl+2tBcCCGjlWrH9Msw= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - rados: clients: - client.2 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:20:47 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T20:00:46.826 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa; will attempt to use it 2024-04-10T20:00:46.827 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks 2024-04-10T20:00:46.827 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T20:00:46.829 INFO:teuthology.task.internal:Saving configuration 2024-04-10T20:00:46.841 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T20:00:46.852 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:00:46.888 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:55:05.951086', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHoJThUGdtFA9y8J50DIpJZrXDiHovtyIB+U9o9qf+Btdp8AvW353V0ElnF/bo01qbuR610FMF3B67WwggfI/6k='} 2024-04-10T20:00:46.924 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:55:05.949676', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2XvNrpWdbEZ+0vSPo/qWhsqOSDd4WxfCt2DKfs1Gj0f8TTUCMqpWVFtutnjdAGpH6wSLOvOhHXazf8SWOh0Sc='} 2024-04-10T20:00:46.960 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:55:05.947985', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB+woxYWjV74JfpnRSuShuFxv2xTDMaFxtFWjZgYky/hUGaHnRftBF8L39c8gbg4EsCkgl+2tBcCCGjlWrH9Msw='} 2024-04-10T20:00:46.960 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T20:00:46.970 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T20:00:46.970 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T20:00:46.970 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T20:00:46.971 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T20:00:47.125 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff33e7eaf70>, signals=[15]) 2024-04-10T20:00:47.125 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T20:00:47.135 INFO:teuthology.task.internal:Opening connections... 2024-04-10T20:00:47.136 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-04-10T20:00:47.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:00:47.222 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-04-10T20:00:47.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:00:47.302 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-04-10T20:00:47.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:00:47.387 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T20:00:47.396 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-04-10T20:00:47.419 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-04-10T20:00:47.419 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-04-10T20:00:47.479 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-04-10T20:00:47.479 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T20:00:47.480 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T20:00:47.481 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-04-10T20:00:47.509 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-04-10T20:00:47.531 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-04-10T20:00:47.531 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-04-10T20:00:47.591 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-04-10T20:00:47.591 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="8" 2024-04-10T20:00:47.591 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-04-10T20:00:47.591 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-04-10T20:00:47.591 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="8" 2024-04-10T20:00:47.591 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el8" 2024-04-10T20:00:47.591 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T20:00:47.591 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-04-10T20:00:47.592 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T20:00:47.592 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-04-10T20:00:47.592 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T20:00:47.592 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T20:00:47.592 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T20:00:47.592 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-04-10T20:00:47.645 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-04-10T20:00:47.667 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-04-10T20:00:47.667 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-04-10T20:00:47.728 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T20:00:47.729 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T20:00:47.730 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-04-10T20:00:47.767 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T20:00:47.778 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T20:00:47.826 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T20:00:47.826 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-04-10T20:00:47.828 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-04-10T20:00:47.830 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-04-10T20:00:47.850 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T20:00:47.858 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T20:00:47.858 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-04-10T20:00:47.889 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-04-10T20:00:47.891 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-04-10T20:00:47.928 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T20:00:48.047 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T20:00:48.063 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T20:00:48.063 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T20:00:48.063 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T20:00:48.064 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T20:00:48.064 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T20:00:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:00:48.085 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-04-10T20:00:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:00:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:00:48.087 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-04-10T20:00:48.088 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-04-10T20:00:48.146 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-489.el8.x86_64 2024-04-10T20:00:48.146 INFO:teuthology.task.kernel:Running kernel on smithi131: 4.18.0-489.el8.x86_64 2024-04-10T20:00:48.146 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum install -y kernel 2024-04-10T20:00:48.147 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-489.el8.x86_64 2024-04-10T20:00:48.148 INFO:teuthology.task.kernel:Running kernel on smithi107: 4.18.0-489.el8.x86_64 2024-04-10T20:00:48.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-04-10T20:00:48.149 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-489.el8.x86_64 2024-04-10T20:00:48.149 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-489.el8.x86_64 2024-04-10T20:00:48.149 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2024-04-10T20:00:52.354 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - AppStream 16 MB/s | 28 MB 00:01 2024-04-10T20:00:52.455 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - AppStream 17 MB/s | 28 MB 00:01 2024-04-10T20:00:59.289 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - BaseOS 5.8 MB/s | 10 MB 00:01 2024-04-10T20:01:45.256 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras 411 B/s | 18 kB 00:44 2024-04-10T20:03:01.185 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - AppStream 219 kB/s | 28 MB 02:10 2024-04-10T20:03:11.161 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - BaseOS 79 kB/s | 10 MB 02:13 2024-04-10T20:03:56.567 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras common packages 60 B/s | 7.7 kB 02:11 2024-04-10T20:03:57.268 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras 409 B/s | 18 kB 00:44 2024-04-10T20:05:16.262 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - BaseOS 81 kB/s | 10 MB 02:10 2024-04-10T20:05:25.008 INFO:teuthology.orchestra.run.smithi131.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T20:05:25.008 INFO:teuthology.orchestra.run.smithi131.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T20:05:25.008 INFO:teuthology.orchestra.run.smithi131.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T20:05:25.008 INFO:teuthology.orchestra.run.smithi131.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T20:06:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras common packages 60 B/s | 7.7 kB 02:12 2024-04-10T20:07:32.836 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:23 2024-04-10T20:07:32.837 INFO:teuthology.orchestra.run.smithi107.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T20:07:32.837 INFO:teuthology.orchestra.run.smithi107.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T20:07:32.837 INFO:teuthology.orchestra.run.smithi107.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T20:07:41.178 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 122 kB/s | 16 MB 02:16 2024-04-10T20:07:44.903 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 513 kB/s | 24 kB 00:00 2024-04-10T20:07:45.044 INFO:teuthology.orchestra.run.smithi131.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T20:07:47.850 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T20:07:47.850 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T20:07:47.975 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout:Install 3 Packages 2024-04-10T20:07:47.976 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T20:07:47.977 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 90 M 2024-04-10T20:07:47.977 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 96 M 2024-04-10T20:07:47.977 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-04-10T20:07:50.379 INFO:teuthology.orchestra.run.smithi131.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 4.7 MB/s | 10 MB 00:02 2024-04-10T20:07:59.011 INFO:teuthology.orchestra.run.smithi131.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 3.4 MB/s | 36 MB 00:10 2024-04-10T20:08:06.920 INFO:teuthology.orchestra.run.smithi131.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 2.3 MB/s | 44 MB 00:18 2024-04-10T20:08:06.921 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-04-10T20:08:06.921 INFO:teuthology.orchestra.run.smithi131.stdout:Total 4.8 MB/s | 90 MB 00:18 2024-04-10T20:08:07.608 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-04-10T20:08:08.043 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-04-10T20:08:08.044 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-04-10T20:08:09.302 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-04-10T20:08:09.335 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-04-10T20:08:12.164 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-04-10T20:08:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:08:15.148 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras 103 B/s | 18 kB 02:57 2024-04-10T20:08:15.615 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 7.7 kB 00:00 2024-04-10T20:08:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:08:16.679 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:08:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:08:20.402 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:08:42.704 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:08:42.729 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:08:43.115 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:08:43.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:08:43.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:08:43.600 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:08:43.600 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T20:08:43.600 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-04-10T20:08:43.600 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:08:43.600 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:08:43.600 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T20:08:43.600 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-04-10T20:08:43.775 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:08:43.992 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:08:43.993 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:08:43.993 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:08:43.993 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:08:43.993 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:08:43.994 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T20:08:43.994 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:08:43.994 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:08:44.206 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:08:44.207 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:08:44.207 DEBUG:teuthology.orchestra.run.smithi131:> sudo rpm -qi grub2-tools 2024-04-10T20:08:44.241 INFO:teuthology.orchestra.run.smithi131.stdout:Name : grub2-tools 2024-04-10T20:08:44.241 INFO:teuthology.orchestra.run.smithi131.stdout:Epoch : 1 2024-04-10T20:08:44.241 INFO:teuthology.orchestra.run.smithi131.stdout:Version : 2.02 2024-04-10T20:08:44.241 INFO:teuthology.orchestra.run.smithi131.stdout:Release : 129.el8 2024-04-10T20:08:44.241 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: x86_64 2024-04-10T20:08:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:08:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:Group : System Environment/Base 2024-04-10T20:08:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:Size : 9265586 2024-04-10T20:08:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:License : GPLv3+ 2024-04-10T20:08:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:08:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:08:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:08:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:08:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:Relocations : (not relocatable) 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout:Packager : CentOS BuildSystem 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout:Vendor : CentOS 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout:Summary : Support tools for GRUB. 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout:Description : 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:08:44.243 INFO:teuthology.orchestra.run.smithi131.stdout:hardware devices. 2024-04-10T20:08:44.244 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T20:08:44.244 INFO:teuthology.orchestra.run.smithi131.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:08:44.245 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:08:44.245 INFO:teuthology.task.kernel:Updating grub on smithi131 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:08:44.245 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:08:45.653 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2024-04-10T20:08:47.762 INFO:teuthology.orchestra.run.smithi131.stderr:done 2024-04-10T20:08:47.764 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2024-04-10T20:08:47.797 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.Z1qGrCZapt 2024-04-10T20:08:47.797 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Z1qGrCZapt 2024-04-10T20:08:47.870 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.Z1qGrCZapt 2024-04-10T20:08:48.019 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.Z1qGrCZapt is 6KB 2024-04-10T20:08:48.068 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.Z1qGrCZapt 2024-04-10T20:08:48.085 DEBUG:teuthology.orchestra.run.smithi131:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:08:48.161 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:08:48.161 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:08:48.161 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:08:48.161 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:08:48.162 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:08:48.318 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2024-04-10T20:09:18.320 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T20:09:18.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T20:09:18.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:09:36.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-04-10T20:09:44.047 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T20:09:44.047 INFO:teuthology.orchestra.run.smithi148.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T20:09:44.047 INFO:teuthology.orchestra.run.smithi148.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T20:09:44.047 INFO:teuthology.orchestra.run.smithi148.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T20:09:45.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T20:09:45.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:10:45.814 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T20:10:57.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T20:10:57.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:10:58.167 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-04-10T20:10:58.475 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T20:10:58.475 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:10:58.475 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-04-10T20:10:58.494 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:10:58.495 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:10:58.495 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:10:58.495 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:10:59.495 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T20:10:59.496 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:10:59.562 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2024-04-10T20:10:59.577 DEBUG:teuthology.parallel:result is None 2024-04-10T20:11:59.381 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 123 kB/s | 16 MB 02:15 2024-04-10T20:12:03.172 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 301 kB/s | 24 kB 00:00 2024-04-10T20:12:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T20:12:06.193 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T20:12:06.193 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T20:12:06.305 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-10T20:12:06.305 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout:Install 3 Packages 2024-04-10T20:12:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T20:12:06.307 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 90 M 2024-04-10T20:12:06.307 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 96 M 2024-04-10T20:12:06.307 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-10T20:13:02.200 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 51 kB/s | 16 MB 05:29 2024-04-10T20:13:05.925 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 381 kB/s | 24 kB 00:00 2024-04-10T20:13:06.065 INFO:teuthology.orchestra.run.smithi107.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T20:13:08.821 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T20:13:08.821 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T20:13:08.933 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-10T20:13:08.935 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-10T20:13:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-04-10T20:13:08.935 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-10T20:13:08.935 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-10T20:13:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T20:13:08.935 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-10T20:13:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T20:13:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T20:13:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T20:13:08.936 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-10T20:13:08.936 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-10T20:13:08.936 INFO:teuthology.orchestra.run.smithi107.stdout:Install 3 Packages 2024-04-10T20:13:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T20:13:08.936 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 90 M 2024-04-10T20:13:08.936 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 96 M 2024-04-10T20:13:08.936 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-10T20:13:13.483 INFO:teuthology.orchestra.run.smithi107.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 2.4 MB/s | 10 MB 00:04 2024-04-10T20:13:34.787 INFO:teuthology.orchestra.run.smithi107.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 1.4 MB/s | 36 MB 00:25 2024-04-10T20:13:36.503 INFO:teuthology.orchestra.run.smithi107.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 1.6 MB/s | 44 MB 00:27 2024-04-10T20:13:36.504 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-10T20:13:36.504 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.3 MB/s | 90 MB 00:27 2024-04-10T20:13:37.186 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-10T20:13:37.613 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-10T20:13:37.613 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-10T20:13:38.848 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-10T20:13:38.879 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-10T20:13:41.692 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-10T20:13:42.254 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:13:45.818 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:13:46.211 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:13:49.660 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:13:49.945 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:14:11.820 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:14:11.844 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:14:12.185 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:14:12.185 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:14:12.185 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:14:12.667 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:14:12.667 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T20:14:12.667 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-10T20:14:12.667 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:14:12.667 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:14:12.667 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T20:14:12.667 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-10T20:14:12.830 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:14:13.049 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:14:13.049 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:14:13.049 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:14:13.049 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:14:13.049 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:14:13.049 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T20:14:13.049 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:14:13.050 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:14:13.259 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:14:13.260 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:14:13.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-04-10T20:14:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-04-10T20:14:13.293 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.02 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 129.el8 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Group : System Environment/Base 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 9265586 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Relocations : (not relocatable) 2024-04-10T20:14:13.294 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : CentOS BuildSystem 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T20:14:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:14:13.296 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:14:13.296 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:14:13.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:14:14.994 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-04-10T20:14:17.080 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-04-10T20:14:17.082 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-04-10T20:14:17.123 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.r03XyD4qC7 2024-04-10T20:14:17.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.r03XyD4qC7 2024-04-10T20:14:17.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.r03XyD4qC7 2024-04-10T20:14:17.315 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.r03XyD4qC7 is 6KB 2024-04-10T20:14:17.337 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.r03XyD4qC7 2024-04-10T20:14:17.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:14:17.428 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:14:17.428 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:14:17.429 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:14:17.429 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:14:17.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:14:17.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-04-10T20:14:20.693 INFO:teuthology.orchestra.run.smithi148.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 2.4 MB/s | 10 MB 00:04 2024-04-10T20:14:25.709 INFO:teuthology.orchestra.run.smithi148.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 3.9 MB/s | 36 MB 00:09 2024-04-10T20:14:38.399 INFO:teuthology.orchestra.run.smithi148.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 2.0 MB/s | 44 MB 00:22 2024-04-10T20:14:38.399 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-10T20:14:38.399 INFO:teuthology.orchestra.run.smithi148.stdout:Total 609 kB/s | 90 MB 02:32 2024-04-10T20:14:39.083 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-10T20:14:39.514 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-10T20:14:39.514 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-10T20:14:40.771 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-10T20:14:40.802 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-10T20:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-10T20:14:44.140 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:14:47.597 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T20:14:47.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-10T20:14:47.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:14:47.777 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:14:48.190 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:14:51.630 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:14:51.904 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:15:13.749 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:15:13.773 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:15:14.109 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:15:14.109 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:15:14.109 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:15:14.595 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:15:14.595 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T20:15:14.595 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-10T20:15:14.596 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:15:14.596 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:15:14.596 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T20:15:14.596 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-10T20:15:14.775 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:15:14.998 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:15:14.998 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:15:14.998 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:15:14.998 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:15:14.998 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:15:14.998 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T20:15:14.998 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:15:14.998 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:15:15.208 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:15:15.209 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:15:15.209 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2024-04-10T20:15:15.242 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2024-04-10T20:15:15.242 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.02 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 129.el8 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Group : System Environment/Base 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 9265586 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:15:15.243 INFO:teuthology.orchestra.run.smithi148.stdout:Relocations : (not relocatable) 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : CentOS BuildSystem 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T20:15:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:15:15.245 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:15:15.245 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:15:15.245 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:15:17.062 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2024-04-10T20:15:19.381 INFO:teuthology.orchestra.run.smithi148.stderr:done 2024-04-10T20:15:19.383 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-04-10T20:15:19.446 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.xWxRSAvlTU 2024-04-10T20:15:19.446 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xWxRSAvlTU 2024-04-10T20:15:19.527 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.xWxRSAvlTU 2024-04-10T20:15:19.691 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.xWxRSAvlTU is 6KB 2024-04-10T20:15:19.740 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.xWxRSAvlTU 2024-04-10T20:15:19.758 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:15:19.823 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:15:19.823 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:15:19.823 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:15:19.823 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:15:19.824 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:15:19.974 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2024-04-10T20:15:22.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-10T20:15:31.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-10T20:15:31.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:15:49.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-10T20:15:49.977 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T20:15:49.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-10T20:15:49.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:16:01.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-10T20:16:01.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:16:01.824 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-04-10T20:16:02.116 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-10T20:16:02.117 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:16:02.117 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-04-10T20:16:02.136 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:16:02.136 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:16:02.136 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:16:02.136 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:16:03.137 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T20:16:03.137 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:16:03.200 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-04-10T20:16:03.213 DEBUG:teuthology.parallel:result is None 2024-04-10T20:16:24.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-04-10T20:16:33.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-10T20:16:33.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:17:33.623 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T20:17:45.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-10T20:17:45.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:17:45.967 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-04-10T20:17:46.258 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-10T20:17:46.258 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:17:46.258 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-04-10T20:17:46.278 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:17:46.278 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:17:46.278 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:17:46.278 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:17:47.279 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T20:17:47.279 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:17:47.369 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2024-04-10T20:17:47.379 DEBUG:teuthology.parallel:result is None 2024-04-10T20:17:47.380 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T20:17:47.388 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T20:17:47.388 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:17:47.393 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:17:47.395 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:17:47.415 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T20:17:47.437 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T20:17:47.478 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T20:17:47.478 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:17:47.481 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:17:47.483 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:17:47.529 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T20:17:47.539 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T20:17:47.539 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:17:47.557 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:17:47.566 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:17:47.599 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:17:47.604 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:17:47.607 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:17:47.611 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:17:47.615 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:17:47.618 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:17:47.619 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T20:17:47.629 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T20:17:47.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:17:47.631 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:17:47.663 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:17:47.692 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T20:17:47.703 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T20:17:47.704 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:17:47.705 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:17:47.732 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:17:47.751 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:17:47.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:17:47.867 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:17:47.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:17:47.998 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-10T20:17:47.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:17:48.066 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:17:48.125 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:17:48.170 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:17:48.265 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:17:48.298 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-10T20:17:48.298 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:17:48.365 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:17:48.422 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:17:48.469 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:17:48.565 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:17:48.601 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-10T20:17:48.601 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:17:48.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-04-10T20:17:48.672 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-04-10T20:17:48.674 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-04-10T20:17:48.754 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:17:48.759 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:17:48.772 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:17:49.215 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T20:17:49.223 INFO:teuthology.task.internal:Starting timer... 2024-04-10T20:17:49.223 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T20:17:49.276 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T20:17:49.313 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T20:17:49.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-04-10T20:17:49.396 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-04-10T20:17:49.397 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2024-04-10T20:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2024-04-10T20:17:49.469 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2024-04-10T20:17:49.538 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2024-04-10T20:17:49.540 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T20:17:49.540 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-04-10T20:17:49.584 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-04-10T20:17:49.584 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2024-04-10T20:17:49.657 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2024-04-10T20:17:49.658 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2024-04-10T20:17:49.685 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2024-04-10T20:17:49.686 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi131.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive'} 2024-04-10T20:17:49.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:17:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:17:49.714 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:17:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:17:49.740 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:17:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:17:49.768 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T20:17:49.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:17:49.796 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:17:49.822 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:17:49.849 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T20:17:49.859 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T20:17:49.865 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T20:17:50.057 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T20:17:50.072 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T20:17:50.073 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi131.front.sepia.ceph.com,smithi148.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T20:33:57.725 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130/ansible_failures.yaml 2024-04-10T20:33:57.728 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T20:33:57.952 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a82e90cfa0bc4ceebe1009b750095bd7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T20:33:57.954 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T20:33:57.962 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T20:33:57.962 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T20:33:58.001 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T20:33:58.083 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T20:33:58.147 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T20:33:58.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:33:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:33:58.247 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:33:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:33:58.278 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:33:58.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:33:58.309 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T20:33:58.317 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T20:33:58.350 INFO:teuthology.task.internal:Duration was 969.127517 seconds 2024-04-10T20:33:58.351 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T20:33:58.385 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T20:33:58.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T20:33:58.387 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T20:33:58.390 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T20:33:58.435 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:33:58.437 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:33:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:33:58.768 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T20:33:58.768 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-04-10T20:33:58.768 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T20:33:58.791 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-04-10T20:33:58.792 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T20:33:58.818 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-04-10T20:33:58.818 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T20:33:58.842 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T20:33:58.842 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T20:33:58.845 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T20:33:58.860 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T20:33:58.925 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T20:33:58.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T20:33:58.927 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T20:33:58.929 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T20:33:59.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T20:33:59.036 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T20:33:59.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T20:33:59.062 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T20:33:59.089 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T20:33:59.117 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T20:33:59.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:33:59.128 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:33:59.132 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:33:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-04-10T20:33:59.156 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-04-10T20:33:59.184 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-04-10T20:33:59.206 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:33:59.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:33:59.231 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:33:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:33:59.247 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:33:59.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:33:59.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T20:33:59.273 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T20:33:59.274 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130/remote/smithi107 2024-04-10T20:33:59.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T20:33:59.323 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130/remote/smithi131 2024-04-10T20:33:59.324 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T20:33:59.376 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130/remote/smithi148 2024-04-10T20:33:59.377 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T20:33:59.436 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T20:33:59.436 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T20:33:59.438 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T20:33:59.440 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T20:33:59.458 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T20:33:59.468 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T20:33:59.468 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T20:33:59.485 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T20:33:59.485 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T20:33:59.495 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T20:33:59.498 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T20:33:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 20:33 /home/ubuntu/cephtest 2024-04-10T20:33:59.512 INFO:teuthology.orchestra.run.smithi131.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 20:33 /home/ubuntu/cephtest 2024-04-10T20:33:59.515 INFO:teuthology.orchestra.run.smithi148.stdout: 525354 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 20:33 /home/ubuntu/cephtest 2024-04-10T20:33:59.516 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T20:33:59.525 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T20:33:59.587 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T20:33:59.622 DEBUG:teuthology.nuke:shortname: smithi107 2024-04-10T20:33:59.623 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:33:59.671 DEBUG:teuthology.nuke:shortname: smithi131 2024-04-10T20:33:59.671 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:33:59.699 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:55:05.951086', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHoJThUGdtFA9y8J50DIpJZrXDiHovtyIB+U9o9qf+Btdp8AvW353V0ElnF/bo01qbuR610FMF3B67WwggfI/6k='} 2024-04-10T20:33:59.718 DEBUG:teuthology.nuke:shortname: smithi148 2024-04-10T20:33:59.718 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:33:59.732 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:55:05.949676', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2XvNrpWdbEZ+0vSPo/qWhsqOSDd4WxfCt2DKfs1Gj0f8TTUCMqpWVFtutnjdAGpH6wSLOvOhHXazf8SWOh0Sc='} 2024-04-10T20:33:59.760 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:55:05.947985', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB+woxYWjV74JfpnRSuShuFxv2xTDMaFxtFWjZgYky/hUGaHnRftBF8L39c8gbg4EsCkgl+2tBcCCGjlWrH9Msw='} 2024-04-10T20:33:59.813 INFO:teuthology.orchestra.console.smithi107:Power off 2024-04-10T20:33:59.813 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T20:33:59.829 INFO:teuthology.orchestra.console.smithi131:Power off 2024-04-10T20:33:59.830 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T20:33:59.848 INFO:teuthology.orchestra.console.smithi148:Power off 2024-04-10T20:33:59.848 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T20:33:59.866 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-04-10T20:33:59.866 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:33:59.880 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-04-10T20:33:59.880 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:33:59.900 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2024-04-10T20:33:59.901 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:33:59.917 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-10T20:33:59.931 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-04-10T20:33:59.955 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-04-10T20:34:03.918 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:34:03.932 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:34:03.956 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:34:04.029 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-10T20:34:04.041 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-04-10T20:34:04.066 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-04-10T20:34:08.034 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:34:08.043 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:34:08.066 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:34:08.145 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-04-10T20:34:08.152 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-04-10T20:34:08.175 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2024-04-10T20:34:08.246 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-04-10T20:34:08.276 INFO:teuthology.orchestra.console.smithi148:Power off completed 2024-04-10T20:34:12.154 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:34:12.264 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-04-10T20:34:12.365 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-04-10T20:34:12.471 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} duration: 969.1275174617767 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a82e90cfa0bc4ceebe1009b750095bd7 status: dead success: false 2024-04-10T20:34:12.471 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T20:34:12.787 INFO:teuthology.run:DEAD