2024-04-10T19:15:30.346 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T19:15:30.346 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038 --verbose --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} -- /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038/orig.config.yaml 2024-04-10T19:15:30.388 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T19:15:30.537 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038 branch: wip-yuri10-testing-2024-03-24-1159 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} email: yweinste@redhat.com first_in_suite: false job_id: '7651038' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-03-24-1159 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum mon_bind_msgr2: false sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri10-testing-2024-03-24-1159 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 8148 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-03-24-1159 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPO/nZcPXZ6P8nas7Xu6IRNQMGM5+3/UOlhlw2GFM5A0zqqjNfYw2JAKTkWSbFO0xUQAVuui83o/EyO0G0mdUc= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAvCTOvh5gT6sfAfB1s4ErS8X9ROmASAxPpGvp18ok9s0bfwhiVIbSn56qLeo6St5vXHhNTPmH3wWCzj24dpTIY= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhbIj6lokm+bJylZbB6CgPCIOaKz0OJBHlRUJRjKe/1Bk6K6/fuSZBdu/p4a8gJ6W/zmuUErcSP4t9vnc0IOfA= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:20:47 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T19:15:30.537 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa; will attempt to use it 2024-04-10T19:15:30.538 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks 2024-04-10T19:15:30.538 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T19:15:30.540 INFO:teuthology.task.internal:Saving configuration 2024-04-10T19:15:30.556 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T19:15:30.563 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:15:30.662 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:08:25.843162', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPO/nZcPXZ6P8nas7Xu6IRNQMGM5+3/UOlhlw2GFM5A0zqqjNfYw2JAKTkWSbFO0xUQAVuui83o/EyO0G0mdUc='} 2024-04-10T19:15:30.754 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:08:25.841589', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAvCTOvh5gT6sfAfB1s4ErS8X9ROmASAxPpGvp18ok9s0bfwhiVIbSn56qLeo6St5vXHhNTPmH3wWCzj24dpTIY='} 2024-04-10T19:15:30.845 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:08:25.839746', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhbIj6lokm+bJylZbB6CgPCIOaKz0OJBHlRUJRjKe/1Bk6K6/fuSZBdu/p4a8gJ6W/zmuUErcSP4t9vnc0IOfA='} 2024-04-10T19:15:30.845 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T19:15:30.852 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T19:15:30.852 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T19:15:30.852 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T19:15:30.852 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T19:15:31.163 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fef66547f70>, signals=[15]) 2024-04-10T19:15:31.164 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T19:15:31.171 INFO:teuthology.task.internal:Opening connections... 2024-04-10T19:15:31.171 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-04-10T19:15:31.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:15:31.320 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-04-10T19:15:31.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:15:31.481 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-04-10T19:15:31.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:15:31.622 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T19:15:31.628 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-04-10T19:15:31.651 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-04-10T19:15:31.651 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8" 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8" 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8" 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T19:15:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-04-10T19:15:31.715 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T19:15:31.715 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T19:15:31.715 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T19:15:31.715 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-04-10T19:15:31.811 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-04-10T19:15:31.833 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-04-10T19:15:31.833 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-04-10T19:15:31.895 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-04-10T19:15:31.895 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8" 2024-04-10T19:15:31.895 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-04-10T19:15:31.895 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-04-10T19:15:31.895 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8" 2024-04-10T19:15:31.895 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2024-04-10T19:15:31.895 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T19:15:31.895 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-04-10T19:15:31.895 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T19:15:31.896 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-04-10T19:15:31.896 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T19:15:31.896 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T19:15:31.896 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T19:15:31.896 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-04-10T19:15:31.995 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-04-10T19:15:32.016 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-04-10T19:15:32.017 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-04-10T19:15:32.077 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-04-10T19:15:32.077 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="8" 2024-04-10T19:15:32.077 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="8" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el8" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T19:15:32.078 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T19:15:32.078 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-04-10T19:15:32.178 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T19:15:32.188 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T19:15:32.195 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T19:15:32.196 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-04-10T19:15:32.198 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-04-10T19:15:32.201 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-04-10T19:15:32.221 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T19:15:32.227 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T19:15:32.227 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-04-10T19:15:32.259 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-04-10T19:15:32.262 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-04-10T19:15:32.299 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T19:15:32.609 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T19:15:32.625 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T19:15:32.625 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T19:15:32.625 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T19:15:32.625 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T19:15:32.626 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T19:15:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:15:32.647 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-04-10T19:15:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:15:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:15:32.648 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-10T19:15:32.649 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-04-10T19:15:32.708 INFO:teuthology.orchestra.run.smithi196.stdout:4.18.0-489.el8.x86_64 2024-04-10T19:15:32.708 INFO:teuthology.task.kernel:Running kernel on smithi196: 4.18.0-489.el8.x86_64 2024-04-10T19:15:32.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-04-10T19:15:32.710 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-489.el8.x86_64 2024-04-10T19:15:32.710 INFO:teuthology.task.kernel:Running kernel on smithi086: 4.18.0-489.el8.x86_64 2024-04-10T19:15:32.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-04-10T19:15:32.711 INFO:teuthology.orchestra.run.smithi032.stdout:4.18.0-489.el8.x86_64 2024-04-10T19:15:32.711 INFO:teuthology.task.kernel:Running kernel on smithi032: 4.18.0-489.el8.x86_64 2024-04-10T19:15:32.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-04-10T19:15:39.617 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - AppStream 6.6 MB/s | 28 MB 00:04 2024-04-10T19:15:41.455 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 8 - AppStream 4.5 MB/s | 28 MB 00:06 2024-04-10T19:17:50.003 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - AppStream 212 kB/s | 28 MB 02:14 2024-04-10T19:17:55.220 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - BaseOS 81 kB/s | 10 MB 02:10 2024-04-10T19:17:59.128 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - BaseOS 2.8 MB/s | 10 MB 00:03 2024-04-10T19:18:01.528 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras 27 kB/s | 18 kB 00:00 2024-04-10T19:18:01.699 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 8 - BaseOS 78 kB/s | 10 MB 02:15 2024-04-10T19:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 8 - Extras 17 kB/s | 18 kB 00:01 2024-04-10T19:18:46.134 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras common packages 177 B/s | 7.7 kB 00:44 2024-04-10T19:20:09.473 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras 137 B/s | 18 kB 02:12 2024-04-10T19:20:14.591 INFO:teuthology.orchestra.run.smithi032.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T19:20:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T19:20:14.592 INFO:teuthology.orchestra.run.smithi032.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T19:20:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T19:20:16.071 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-10T19:20:20.044 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 583 kB/s | 24 kB 00:00 2024-04-10T19:20:20.194 INFO:teuthology.orchestra.run.smithi032.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:20:23.099 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:20:23.099 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:20:23.199 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-10T19:20:23.200 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-10T19:20:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-10T19:20:23.200 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-10T19:20:23.200 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-10T19:20:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:20:23.201 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-10T19:20:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:20:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:20:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-10T19:20:23.201 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-10T19:20:23.201 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-10T19:20:23.201 INFO:teuthology.orchestra.run.smithi032.stdout:Install 3 Packages 2024-04-10T19:20:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-10T19:20:23.202 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 90 M 2024-04-10T19:20:23.202 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 96 M 2024-04-10T19:20:23.202 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-10T19:20:26.153 INFO:teuthology.orchestra.run.smithi032.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 7.7 MB/s | 10 MB 00:01 2024-04-10T19:20:27.372 INFO:teuthology.orchestra.run.smithi032.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 14 MB/s | 36 MB 00:02 2024-04-10T19:20:27.870 INFO:teuthology.orchestra.run.smithi032.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 14 MB/s | 44 MB 00:03 2024-04-10T19:20:27.871 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:20:27.871 INFO:teuthology.orchestra.run.smithi032.stdout:Total 19 MB/s | 90 MB 00:04 2024-04-10T19:20:28.584 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-10T19:20:29.043 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-10T19:20:29.043 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-10T19:20:30.375 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-10T19:20:30.408 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-10T19:20:33.364 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-10T19:20:33.860 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:20:37.565 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:20:37.944 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:20:41.633 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:20:41.921 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:20:50.236 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras common packages 194 B/s | 7.7 kB 00:40 2024-04-10T19:21:03.320 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 8 - Extras common packages 44 B/s | 7.7 kB 02:58 2024-04-10T19:21:05.345 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:21:05.370 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:21:05.737 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:21:05.737 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:21:05.737 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:21:06.726 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:21:06.726 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-10T19:21:06.726 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-10T19:21:06.726 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T19:21:06.726 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T19:21:06.726 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-10T19:21:06.726 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-10T19:21:06.871 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:21:07.097 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:21:07.098 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:21:07.098 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T19:21:07.098 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T19:21:07.098 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T19:21:07.098 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T19:21:07.098 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T19:21:07.099 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:21:07.320 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:21:07.321 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:21:07.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-04-10T19:21:07.355 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-04-10T19:21:07.355 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-04-10T19:21:07.355 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.02 2024-04-10T19:21:07.355 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 129.el8 2024-04-10T19:21:07.355 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-04-10T19:21:07.355 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T19:21:07.355 INFO:teuthology.orchestra.run.smithi032.stdout:Group : System Environment/Base 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 9265586 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Relocations : (not relocatable) 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : CentOS BuildSystem 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-10T19:21:07.356 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T19:21:07.357 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T19:21:07.357 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T19:21:07.357 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-04-10T19:21:07.357 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-10T19:21:07.357 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T19:21:07.357 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T19:21:07.358 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 4.18.0-552.el8.x86_64 2024-04-10T19:21:07.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T19:21:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-04-10T19:21:10.771 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-04-10T19:21:10.773 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-04-10T19:21:10.808 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.V0ce3xs01A 2024-04-10T19:21:10.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.V0ce3xs01A 2024-04-10T19:21:10.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.V0ce3xs01A 2024-04-10T19:21:11.022 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.V0ce3xs01A is 6KB 2024-04-10T19:21:11.070 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.V0ce3xs01A 2024-04-10T19:21:11.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T19:21:11.155 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T19:21:11.155 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T19:21:11.155 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T19:21:11.155 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T19:21:11.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T19:21:11.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-04-10T19:21:41.311 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T19:21:41.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-10T19:21:41.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:22:20.685 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:30 2024-04-10T19:22:20.685 INFO:teuthology.orchestra.run.smithi086.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T19:22:20.686 INFO:teuthology.orchestra.run.smithi086.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T19:22:20.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T19:22:23.228 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 16 MB 00:02 2024-04-10T19:22:26.945 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 396 kB/s | 24 kB 00:00 2024-04-10T19:22:27.090 INFO:teuthology.orchestra.run.smithi086.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:22:29.933 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:22:29.934 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:22:30.051 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-10T19:22:30.052 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-10T19:22:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-04-10T19:22:30.052 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout:Install 3 Packages 2024-04-10T19:22:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-10T19:22:30.054 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 90 M 2024-04-10T19:22:30.054 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 96 M 2024-04-10T19:22:30.054 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-10T19:22:31.763 INFO:teuthology.orchestra.run.smithi196.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T19:22:31.763 INFO:teuthology.orchestra.run.smithi196.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T19:22:31.763 INFO:teuthology.orchestra.run.smithi196.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T19:22:31.764 INFO:teuthology.orchestra.run.smithi196.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T19:22:41.330 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T19:22:50.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-10T19:22:50.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:22:50.669 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-10T19:22:50.959 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-10T19:22:50.959 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T19:22:50.959 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-10T19:22:50.979 INFO:teuthology.orchestra.run.smithi032.stdout:4.18.0-552.el8.x86_64 2024-04-10T19:22:50.979 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T19:22:50.979 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T19:22:50.979 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T19:22:51.980 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T19:22:51.981 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T19:22:52.059 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-04-10T19:22:52.075 DEBUG:teuthology.parallel:result is None 2024-04-10T19:24:40.216 INFO:teuthology.orchestra.run.smithi086.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 8.9 MB/s | 10 MB 00:01 2024-04-10T19:24:41.274 INFO:teuthology.orchestra.run.smithi086.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-04-10T19:24:41.858 INFO:teuthology.orchestra.run.smithi086.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 16 MB/s | 44 MB 00:02 2024-04-10T19:24:41.858 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:24:41.858 INFO:teuthology.orchestra.run.smithi086.stdout:Total 703 kB/s | 90 MB 02:11 2024-04-10T19:24:42.549 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-10T19:24:42.986 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-10T19:24:42.987 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-10T19:24:44.259 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-10T19:24:44.293 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-10T19:24:44.428 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 126 kB/s | 16 MB 02:12 2024-04-10T19:24:47.129 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-10T19:24:47.719 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:24:48.159 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 395 kB/s | 24 kB 00:00 2024-04-10T19:24:48.300 INFO:teuthology.orchestra.run.smithi196.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:24:51.114 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:24:51.115 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:24:51.247 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout:Install 3 Packages 2024-04-10T19:24:51.248 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-10T19:24:51.249 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 90 M 2024-04-10T19:24:51.249 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 96 M 2024-04-10T19:24:51.249 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-10T19:24:51.281 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:24:51.677 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:24:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:24:55.426 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:25:17.372 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:25:17.395 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:25:17.761 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:25:17.761 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:25:17.761 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:25:18.253 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:25:18.253 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-10T19:25:18.253 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-10T19:25:18.253 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T19:25:18.253 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T19:25:18.253 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-10T19:25:18.253 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-10T19:25:18.431 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:25:18.648 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:25:18.648 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:25:18.648 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T19:25:18.649 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T19:25:18.649 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T19:25:18.649 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T19:25:18.649 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T19:25:18.649 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:25:18.860 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:25:18.860 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:25:18.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.02 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 129.el8 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:Group : System Environment/Base 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 9265586 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-04-10T19:25:18.894 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:Relocations : (not relocatable) 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : CentOS BuildSystem 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-04-10T19:25:18.895 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-10T19:25:18.896 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T19:25:18.896 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T19:25:18.896 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 4.18.0-552.el8.x86_64 2024-04-10T19:25:18.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T19:25:20.617 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-04-10T19:25:23.038 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-04-10T19:25:23.049 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-04-10T19:25:23.113 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.vfpRkLkLGe 2024-04-10T19:25:23.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vfpRkLkLGe 2024-04-10T19:25:23.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.vfpRkLkLGe 2024-04-10T19:25:23.341 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.vfpRkLkLGe is 6KB 2024-04-10T19:25:23.389 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.vfpRkLkLGe 2024-04-10T19:25:23.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T19:25:23.480 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T19:25:23.480 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T19:25:23.480 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T19:25:23.480 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T19:25:23.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T19:25:23.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-04-10T19:25:53.646 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T19:25:53.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-10T19:25:53.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:26:12.002 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-04-10T19:26:21.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-10T19:26:21.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:26:24.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-04-10T19:26:36.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-10T19:26:36.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:27:04.714 INFO:teuthology.orchestra.run.smithi196.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 4.5 MB/s | 10 MB 00:02 2024-04-10T19:27:06.538 INFO:teuthology.orchestra.run.smithi196.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 11 MB/s | 44 MB 00:04 2024-04-10T19:27:07.722 INFO:teuthology.orchestra.run.smithi196.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 6.9 MB/s | 36 MB 00:05 2024-04-10T19:27:07.722 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:27:07.722 INFO:teuthology.orchestra.run.smithi196.stdout:Total 679 kB/s | 90 MB 02:16 2024-04-10T19:27:08.122 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-04-10T19:27:08.387 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-10T19:27:08.387 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T19:27:08.387 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-04-10T19:27:08.404 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-552.el8.x86_64 2024-04-10T19:27:08.404 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T19:27:08.404 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T19:27:08.404 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T19:27:08.410 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-10T19:27:08.878 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-10T19:27:08.900 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-10T19:27:09.405 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T19:27:09.405 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T19:27:09.479 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-04-10T19:27:09.491 DEBUG:teuthology.parallel:result is None 2024-04-10T19:27:10.119 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-10T19:27:10.153 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-10T19:27:12.963 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-10T19:27:13.464 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:27:17.044 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:27:17.455 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:27:20.907 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:27:21.178 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:27:42.950 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:27:42.973 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:27:43.322 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:27:43.322 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:27:43.322 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:27:44.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:27:44.118 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-10T19:27:44.118 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-10T19:27:44.118 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T19:27:44.118 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T19:27:44.118 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-10T19:27:44.118 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-10T19:27:44.245 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:27:44.468 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:27:44.468 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:27:44.468 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T19:27:44.468 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T19:27:44.468 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T19:27:44.469 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T19:27:44.469 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T19:27:44.469 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:27:44.681 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:27:44.681 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:27:44.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-04-10T19:27:44.715 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-04-10T19:27:44.715 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-04-10T19:27:44.715 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.02 2024-04-10T19:27:44.715 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 129.el8 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Group : System Environment/Base 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 9265586 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Relocations : (not relocatable) 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : CentOS BuildSystem 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T19:27:44.716 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-04-10T19:27:44.717 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-04-10T19:27:44.717 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-10T19:27:44.717 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T19:27:44.717 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T19:27:44.717 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T19:27:44.717 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-04-10T19:27:44.717 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-10T19:27:44.717 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T19:27:44.718 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T19:27:44.718 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 4.18.0-552.el8.x86_64 2024-04-10T19:27:44.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T19:27:46.082 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-04-10T19:27:48.661 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-04-10T19:27:48.662 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-04-10T19:27:48.686 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.cDxpiTakGh 2024-04-10T19:27:48.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cDxpiTakGh 2024-04-10T19:27:48.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.cDxpiTakGh 2024-04-10T19:27:48.904 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.cDxpiTakGh is 6KB 2024-04-10T19:27:48.953 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.cDxpiTakGh 2024-04-10T19:27:48.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T19:27:49.058 INFO:teuthology.orchestra.run.smithi196.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T19:27:49.058 INFO:teuthology.orchestra.run.smithi196.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T19:27:49.058 INFO:teuthology.orchestra.run.smithi196.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T19:27:49.058 INFO:teuthology.orchestra.run.smithi196.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T19:27:49.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T19:27:49.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-04-10T19:28:19.173 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T19:28:19.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-10T19:28:19.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:28:37.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-04-10T19:28:46.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-10T19:28:46.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:28:49.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-04-10T19:29:01.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-10T19:29:01.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:29:20.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-04-10T19:29:35.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-10T19:29:35.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:29:36.104 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-04-10T19:29:36.602 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-10T19:29:36.602 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T19:29:36.603 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-04-10T19:29:36.620 INFO:teuthology.orchestra.run.smithi196.stdout:4.18.0-552.el8.x86_64 2024-04-10T19:29:36.620 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T19:29:36.620 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T19:29:36.621 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T19:29:37.622 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T19:29:37.622 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T19:29:37.690 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-04-10T19:29:37.705 DEBUG:teuthology.parallel:result is None 2024-04-10T19:29:37.706 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T19:29:37.718 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T19:29:37.718 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T19:29:37.722 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T19:29:37.726 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T19:29:37.746 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T19:29:37.753 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T19:29:37.763 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T19:29:37.763 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T19:29:37.785 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T19:29:37.788 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T19:29:37.835 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T19:29:37.843 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T19:29:37.843 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T19:29:37.869 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T19:29:37.872 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T19:29:37.910 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:29:37.921 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:29:37.922 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:29:37.924 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:29:37.931 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:29:37.934 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:29:37.936 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T19:29:37.943 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T19:29:37.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T19:29:37.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T19:29:37.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T19:29:37.999 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T19:29:38.008 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T19:29:38.008 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T19:29:38.040 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T19:29:38.042 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T19:29:38.061 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:29:38.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:29:38.210 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:29:38.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:29:38.300 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-10T19:29:38.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T19:29:38.369 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:29:38.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:29:38.471 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:29:38.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:29:38.601 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-10T19:29:38.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T19:29:38.671 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:29:38.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:29:38.782 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:29:38.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:29:38.872 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-10T19:29:38.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T19:29:38.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-04-10T19:29:38.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-04-10T19:29:38.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-04-10T19:29:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:29:39.020 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:29:39.048 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:29:39.392 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T19:29:39.400 INFO:teuthology.task.internal:Starting timer... 2024-04-10T19:29:39.400 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T19:29:39.410 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T19:29:39.418 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T19:29:39.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-04-10T19:29:39.491 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-04-10T19:29:39.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-04-10T19:29:39.570 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-04-10T19:29:39.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-04-10T19:29:39.650 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-04-10T19:29:39.652 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T19:29:39.652 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-04-10T19:29:39.688 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-04-10T19:29:39.689 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-04-10T19:29:39.743 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-04-10T19:29:39.743 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-04-10T19:29:39.770 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-04-10T19:29:39.770 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-04-10T19:29:39.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:29:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:29:39.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:29:39.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:29:39.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:29:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:29:39.857 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T19:29:39.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-04-10T19:29:39.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-04-10T19:29:39.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-04-10T19:29:39.940 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T19:29:39.949 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T19:29:39.954 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T19:29:40.169 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T19:29:40.183 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T19:29:40.184 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi086.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T19:47:49.395 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038/ansible_failures.yaml 2024-04-10T19:47:49.397 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T19:47:49.601 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3d44a73d81f04b8e863d5dbd10089213 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T19:47:49.602 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T19:47:49.611 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T19:47:49.611 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T19:47:49.629 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T19:47:49.723 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T19:47:49.789 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T19:47:49.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:47:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:47:49.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:47:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:47:49.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:47:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:47:49.950 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T19:47:49.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T19:47:49.966 INFO:teuthology.task.internal:Duration was 1090.565420 seconds 2024-04-10T19:47:49.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T19:47:49.974 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T19:47:49.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T19:47:49.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T19:47:49.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T19:47:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:47:50.028 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:47:50.038 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:47:50.399 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T19:47:50.399 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-04-10T19:47:50.400 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T19:47:50.424 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-04-10T19:47:50.424 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T19:47:50.451 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-04-10T19:47:50.451 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T19:47:50.474 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T19:47:50.474 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T19:47:50.476 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T19:47:50.494 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T19:47:50.553 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T19:47:50.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T19:47:50.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T19:47:50.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T19:47:50.655 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T19:47:50.664 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T19:47:50.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T19:47:50.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T19:47:50.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T19:47:50.745 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T19:47:50.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:47:50.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:47:50.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:47:50.781 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-04-10T19:47:50.786 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-04-10T19:47:50.812 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-04-10T19:47:50.831 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:47:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:47:50.859 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:47:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:47:50.877 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:47:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:47:50.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T19:47:50.902 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T19:47:50.904 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038/remote/smithi032 2024-04-10T19:47:50.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T19:47:50.970 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038/remote/smithi086 2024-04-10T19:47:50.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T19:47:51.015 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038/remote/smithi196 2024-04-10T19:47:51.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T19:47:51.065 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T19:47:51.065 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T19:47:51.067 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T19:47:51.070 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T19:47:51.116 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T19:47:51.126 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T19:47:51.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T19:47:51.133 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T19:47:51.134 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T19:47:51.136 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T19:47:51.138 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T19:47:51.151 INFO:teuthology.orchestra.run.smithi032.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 19:47 /home/ubuntu/cephtest 2024-04-10T19:47:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 19:47 /home/ubuntu/cephtest 2024-04-10T19:47:51.172 INFO:teuthology.orchestra.run.smithi196.stdout: 525355 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 19:47 /home/ubuntu/cephtest 2024-04-10T19:47:51.174 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T19:47:51.182 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T19:47:51.199 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T19:47:51.247 DEBUG:teuthology.nuke:shortname: smithi032 2024-04-10T19:47:51.248 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:47:51.312 DEBUG:teuthology.nuke:shortname: smithi086 2024-04-10T19:47:51.313 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:47:51.361 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:08:25.843162', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPO/nZcPXZ6P8nas7Xu6IRNQMGM5+3/UOlhlw2GFM5A0zqqjNfYw2JAKTkWSbFO0xUQAVuui83o/EyO0G0mdUc='} 2024-04-10T19:47:51.386 DEBUG:teuthology.nuke:shortname: smithi196 2024-04-10T19:47:51.386 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:47:51.410 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:08:25.841589', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAvCTOvh5gT6sfAfB1s4ErS8X9ROmASAxPpGvp18ok9s0bfwhiVIbSn56qLeo6St5vXHhNTPmH3wWCzj24dpTIY='} 2024-04-10T19:47:51.457 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:08:25.839746', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhbIj6lokm+bJylZbB6CgPCIOaKz0OJBHlRUJRjKe/1Bk6K6/fuSZBdu/p4a8gJ6W/zmuUErcSP4t9vnc0IOfA='} 2024-04-10T19:47:51.513 INFO:teuthology.orchestra.console.smithi032:Power off 2024-04-10T19:47:51.513 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T19:47:51.553 INFO:teuthology.orchestra.console.smithi086:Power off 2024-04-10T19:47:51.554 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T19:47:51.571 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-04-10T19:47:51.571 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:47:51.588 INFO:teuthology.orchestra.console.smithi196:Power off 2024-04-10T19:47:51.588 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T19:47:51.613 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-04-10T19:47:51.613 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:47:51.631 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-10T19:47:51.651 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-04-10T19:47:51.651 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:47:51.672 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-04-10T19:47:51.694 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-04-10T19:47:55.632 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:47:55.673 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:47:55.695 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:47:55.742 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-10T19:47:55.782 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-04-10T19:47:55.805 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-04-10T19:47:59.747 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:47:59.784 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:47:59.806 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:47:59.860 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-04-10T19:47:59.895 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-04-10T19:47:59.916 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-04-10T19:47:59.961 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-04-10T19:47:59.996 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-04-10T19:48:03.920 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:48:04.031 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-04-10T19:48:04.132 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-04-10T19:48:04.238 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} duration: 1090.5654196739197 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3d44a73d81f04b8e863d5dbd10089213 status: dead success: false 2024-04-10T19:48:04.238 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T19:48:04.337 INFO:teuthology.run:DEAD