2024-04-10T19:45:51.841 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T19:45:51.841 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} --name yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094 -- /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094/orig.config.yaml 2024-04-10T19:45:51.880 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T19:45:51.974 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094 branch: wip-yuri10-testing-2024-03-24-1159 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} email: yweinste@redhat.com first_in_suite: false job_id: '7651094' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'rbd object class functional tests ' name: yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 - volumes: size: 50 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-03-24-1159 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 filestore queue throttle high multiple: 2 filestore queue throttle max multiple: 10 journal throttle high multiple: 2 journal throttle max multiple: 10 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 9 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-yuri10-testing-2024-03-24-1159 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 8148 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-03-24-1159 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJo7UEcUn7Zq95c1nc7PwdofWyPtmDUSPjfDf36LKarCV/ajFh7Z5IpYF1yYZe6EnwzdouwiRG4EL0jJfXtUv6o= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIJzLfBkZvWg/mLkeY2xLDdUL71j6gEZjMB2CtN/KIWOwrzBHrNmq/zs0Ad5Bi2PddtglvcBKiDFGYkX6N37gdg= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZ5gdHLm+ka8Jj1+1GQObB3shjvBH94uu6BMKwdkH/xDz9wEOQLGYWeKuc4Bwbl7r7uG2NeqmhhglHiKDHIt5M= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 3 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.2: - ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:20:47 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T19:45:51.974 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa; will attempt to use it 2024-04-10T19:45:51.975 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks 2024-04-10T19:45:51.975 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T19:45:51.977 INFO:teuthology.task.internal:Saving configuration 2024-04-10T19:45:51.990 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T19:45:51.996 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:45:52.035 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:39:33.488208', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJo7UEcUn7Zq95c1nc7PwdofWyPtmDUSPjfDf36LKarCV/ajFh7Z5IpYF1yYZe6EnwzdouwiRG4EL0jJfXtUv6o='} 2024-04-10T19:45:52.069 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:39:33.489548', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIJzLfBkZvWg/mLkeY2xLDdUL71j6gEZjMB2CtN/KIWOwrzBHrNmq/zs0Ad5Bi2PddtglvcBKiDFGYkX6N37gdg='} 2024-04-10T19:45:52.109 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:39:33.486689', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZ5gdHLm+ka8Jj1+1GQObB3shjvBH94uu6BMKwdkH/xDz9wEOQLGYWeKuc4Bwbl7r7uG2NeqmhhglHiKDHIt5M='} 2024-04-10T19:45:52.109 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T19:45:52.115 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T19:45:52.115 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T19:45:52.115 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T19:45:52.115 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T19:45:52.257 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbff7743160>, signals=[15]) 2024-04-10T19:45:52.257 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T19:45:52.264 INFO:teuthology.task.internal:Opening connections... 2024-04-10T19:45:52.264 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-04-10T19:45:52.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:45:52.355 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-04-10T19:45:52.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:45:52.488 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-04-10T19:45:52.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:45:52.575 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T19:45:52.582 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-04-10T19:45:52.605 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-04-10T19:45:52.605 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-04-10T19:45:52.668 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-04-10T19:45:52.668 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="8" 2024-04-10T19:45:52.668 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-04-10T19:45:52.668 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-04-10T19:45:52.668 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="8" 2024-04-10T19:45:52.668 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el8" 2024-04-10T19:45:52.669 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T19:45:52.669 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-04-10T19:45:52.669 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T19:45:52.669 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-04-10T19:45:52.669 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T19:45:52.669 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T19:45:52.669 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T19:45:52.670 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-04-10T19:45:52.708 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-04-10T19:45:52.731 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-04-10T19:45:52.731 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-04-10T19:45:52.791 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-04-10T19:45:52.791 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8" 2024-04-10T19:45:52.791 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-04-10T19:45:52.791 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-04-10T19:45:52.791 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8" 2024-04-10T19:45:52.791 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2024-04-10T19:45:52.791 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T19:45:52.791 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-04-10T19:45:52.791 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T19:45:52.792 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-04-10T19:45:52.792 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T19:45:52.792 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T19:45:52.792 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T19:45:52.792 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-04-10T19:45:52.833 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-04-10T19:45:52.855 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-04-10T19:45:52.856 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-04-10T19:45:52.916 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-04-10T19:45:52.916 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8" 2024-04-10T19:45:52.916 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-04-10T19:45:52.916 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-04-10T19:45:52.916 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8" 2024-04-10T19:45:52.916 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8" 2024-04-10T19:45:52.916 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T19:45:52.916 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-04-10T19:45:52.916 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T19:45:52.917 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-04-10T19:45:52.917 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T19:45:52.917 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T19:45:52.917 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T19:45:52.917 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-04-10T19:45:52.958 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T19:45:52.967 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T19:45:53.054 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T19:45:53.055 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-04-10T19:45:53.057 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-04-10T19:45:53.059 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-04-10T19:45:53.080 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T19:45:53.085 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T19:45:53.085 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-04-10T19:45:53.119 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-04-10T19:45:53.121 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-04-10T19:45:53.152 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T19:45:53.271 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T19:45:53.284 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T19:45:53.285 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T19:45:53.285 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T19:45:53.285 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T19:45:53.286 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T19:45:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:45:53.306 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-10T19:45:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:45:53.307 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-04-10T19:45:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:45:53.308 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-04-10T19:45:53.366 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-489.el8.x86_64 2024-04-10T19:45:53.366 INFO:teuthology.task.kernel:Running kernel on smithi121: 4.18.0-489.el8.x86_64 2024-04-10T19:45:53.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-04-10T19:45:53.369 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-489.el8.x86_64 2024-04-10T19:45:53.369 INFO:teuthology.task.kernel:Running kernel on smithi137: 4.18.0-489.el8.x86_64 2024-04-10T19:45:53.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-04-10T19:45:53.370 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-489.el8.x86_64 2024-04-10T19:45:53.370 INFO:teuthology.task.kernel:Running kernel on smithi038: 4.18.0-489.el8.x86_64 2024-04-10T19:45:53.371 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-04-10T19:45:57.822 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - AppStream 16 MB/s | 28 MB 00:01 2024-04-10T19:45:58.943 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 9.5 MB/s | 28 MB 00:02 2024-04-10T19:46:05.518 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 6.5 MB/s | 10 MB 00:01 2024-04-10T19:48:12.271 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - AppStream 209 kB/s | 28 MB 02:16 2024-04-10T19:48:14.318 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - BaseOS 80 kB/s | 10 MB 02:11 2024-04-10T19:49:02.663 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 104 B/s | 18 kB 02:55 2024-04-10T19:50:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - BaseOS 80 kB/s | 10 MB 02:11 2024-04-10T19:51:12.373 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras 103 B/s | 18 kB 02:56 2024-04-10T19:51:15.812 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 59 B/s | 7.7 kB 02:13 2024-04-10T19:52:40.250 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:24 2024-04-10T19:52:40.251 INFO:teuthology.orchestra.run.smithi137.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T19:52:40.251 INFO:teuthology.orchestra.run.smithi137.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T19:52:40.251 INFO:teuthology.orchestra.run.smithi137.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T19:53:24.714 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras common packages 60 B/s | 7.7 kB 02:12 2024-04-10T19:53:26.856 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras 103 B/s | 18 kB 02:56 2024-04-10T19:53:27.389 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 7.7 kB 00:00 2024-04-10T19:54:52.964 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T19:54:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T19:54:52.965 INFO:teuthology.orchestra.run.smithi038.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T19:54:52.965 INFO:teuthology.orchestra.run.smithi038.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T19:54:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:25 2024-04-10T19:54:53.023 INFO:teuthology.orchestra.run.smithi121.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T19:54:53.023 INFO:teuthology.orchestra.run.smithi121.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T19:54:53.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T19:54:54.615 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2024-04-10T19:55:00.038 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 14 kB/s | 24 kB 00:01 2024-04-10T19:55:00.184 INFO:teuthology.orchestra.run.smithi121.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:55:03.043 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:55:03.043 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:55:03.168 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-10T19:55:03.169 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-10T19:55:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-10T19:55:03.169 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-10T19:55:03.169 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-10T19:55:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:55:03.169 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-10T19:55:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:55:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:55:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-10T19:55:03.170 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-10T19:55:03.170 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-10T19:55:03.170 INFO:teuthology.orchestra.run.smithi121.stdout:Install 3 Packages 2024-04-10T19:55:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-10T19:55:03.171 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 90 M 2024-04-10T19:55:03.171 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 96 M 2024-04-10T19:55:03.171 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-10T19:56:17.792 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 77 kB/s | 16 MB 03:37 2024-04-10T19:56:21.630 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 259 kB/s | 24 kB 00:00 2024-04-10T19:56:21.772 INFO:teuthology.orchestra.run.smithi137.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:56:24.577 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:56:24.577 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:56:24.677 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-10T19:56:24.677 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout:Install 3 Packages 2024-04-10T19:56:24.678 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-10T19:56:24.679 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 90 M 2024-04-10T19:56:24.679 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 96 M 2024-04-10T19:56:24.679 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-10T19:57:07.112 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 124 kB/s | 16 MB 02:14 2024-04-10T19:57:11.052 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 398 kB/s | 24 kB 00:00 2024-04-10T19:57:11.202 INFO:teuthology.orchestra.run.smithi038.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:57:14.181 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:57:14.181 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:57:14.300 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-10T19:57:14.301 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-10T19:57:14.301 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-10T19:57:14.301 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-10T19:57:14.301 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-10T19:57:14.301 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:57:14.302 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-10T19:57:14.302 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:57:14.302 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:57:14.302 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-10T19:57:14.302 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-10T19:57:14.302 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-10T19:57:14.302 INFO:teuthology.orchestra.run.smithi038.stdout:Install 3 Packages 2024-04-10T19:57:14.302 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-10T19:57:14.303 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 90 M 2024-04-10T19:57:14.303 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 96 M 2024-04-10T19:57:14.303 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-10T19:57:17.267 INFO:teuthology.orchestra.run.smithi121.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 2.6 MB/s | 10 MB 00:03 2024-04-10T19:57:20.983 INFO:teuthology.orchestra.run.smithi121.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 4.7 MB/s | 36 MB 00:07 2024-04-10T19:57:36.287 INFO:teuthology.orchestra.run.smithi121.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 1.9 MB/s | 44 MB 00:22 2024-04-10T19:57:36.287 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:57:36.288 INFO:teuthology.orchestra.run.smithi121.stdout:Total 605 kB/s | 90 MB 02:33 2024-04-10T19:57:36.974 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-10T19:57:37.421 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-10T19:57:37.422 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-10T19:57:38.672 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-10T19:57:38.700 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-10T19:57:41.477 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-10T19:57:41.928 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:57:45.467 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:57:45.852 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:57:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:57:49.610 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:58:11.427 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:58:11.450 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:58:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:58:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:58:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:58:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:58:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-10T19:58:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-10T19:58:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T19:58:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T19:58:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-10T19:58:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-10T19:58:12.944 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:58:13.162 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:58:13.162 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:58:13.162 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T19:58:13.162 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T19:58:13.163 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T19:58:13.163 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T19:58:13.163 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T19:58:13.163 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:58:13.373 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:58:13.373 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:58:13.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.02 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 129.el8 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Group : System Environment/Base 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 9265586 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T19:58:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:Relocations : (not relocatable) 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : CentOS BuildSystem 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-10T19:58:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T19:58:13.409 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T19:58:13.409 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 4.18.0-552.el8.x86_64 2024-04-10T19:58:13.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T19:58:14.688 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-04-10T19:58:16.884 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-04-10T19:58:16.887 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-04-10T19:58:16.917 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.2Oj0hLAs2A 2024-04-10T19:58:16.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2Oj0hLAs2A 2024-04-10T19:58:16.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.2Oj0hLAs2A 2024-04-10T19:58:17.134 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.2Oj0hLAs2A is 6KB 2024-04-10T19:58:17.181 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.2Oj0hLAs2A 2024-04-10T19:58:17.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T19:58:17.277 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T19:58:17.277 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T19:58:17.277 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T19:58:17.277 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T19:58:17.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T19:58:17.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-04-10T19:58:39.893 INFO:teuthology.orchestra.run.smithi137.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 3.5 MB/s | 10 MB 00:02 2024-04-10T19:58:47.436 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T19:58:47.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-10T19:58:47.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:58:50.974 INFO:teuthology.orchestra.run.smithi137.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 3.1 MB/s | 44 MB 00:14 2024-04-10T19:59:01.498 INFO:teuthology.orchestra.run.smithi137.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 1.5 MB/s | 36 MB 00:24 2024-04-10T19:59:01.498 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:59:01.498 INFO:teuthology.orchestra.run.smithi137.stdout:Total 591 kB/s | 90 MB 02:36 2024-04-10T19:59:02.183 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-10T19:59:02.643 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-10T19:59:02.643 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-10T19:59:03.909 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-10T19:59:03.946 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-10T19:59:06.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-04-10T19:59:06.803 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-10T19:59:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:59:10.938 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:59:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:59:14.803 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:59:15.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-10T19:59:15.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:59:15.125 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:59:18.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-04-10T19:59:25.646 INFO:teuthology.orchestra.run.smithi038.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 9.2 MB/s | 10 MB 00:01 2024-04-10T19:59:30.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-10T19:59:30.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:59:37.176 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:59:37.200 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:59:37.556 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:59:37.556 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:59:37.557 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:59:38.106 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:59:38.106 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-10T19:59:38.106 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-10T19:59:38.106 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T19:59:38.107 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T19:59:38.107 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-10T19:59:38.107 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-10T19:59:39.034 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:59:39.253 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:59:39.254 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:59:39.254 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T19:59:39.254 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T19:59:39.254 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T19:59:39.254 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T19:59:39.254 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T19:59:39.254 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:59:39.466 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:59:39.467 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:59:39.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-04-10T19:59:39.500 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.02 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 129.el8 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Group : System Environment/Base 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 9265586 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T19:59:39.501 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:Relocations : (not relocatable) 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : CentOS BuildSystem 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T19:59:39.502 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-04-10T19:59:39.503 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-10T19:59:39.503 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T19:59:39.503 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T19:59:39.504 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 4.18.0-552.el8.x86_64 2024-04-10T19:59:39.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T19:59:40.984 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-04-10T19:59:43.317 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-04-10T19:59:43.319 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-04-10T19:59:43.349 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.PGScdyWONT 2024-04-10T19:59:43.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PGScdyWONT 2024-04-10T19:59:43.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.PGScdyWONT 2024-04-10T19:59:43.576 INFO:teuthology.orchestra.run.smithi038.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 1.9 MB/s | 36 MB 00:19 2024-04-10T19:59:43.607 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.PGScdyWONT is 6KB 2024-04-10T19:59:43.654 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.PGScdyWONT 2024-04-10T19:59:43.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T19:59:43.741 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T19:59:43.741 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T19:59:43.742 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T19:59:43.742 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T19:59:43.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T19:59:43.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-04-10T19:59:48.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-04-10T19:59:51.209 INFO:teuthology.orchestra.run.smithi038.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 1.6 MB/s | 44 MB 00:26 2024-04-10T19:59:51.209 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:59:51.209 INFO:teuthology.orchestra.run.smithi038.stdout:Total 590 kB/s | 90 MB 02:36 2024-04-10T19:59:51.927 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-10T19:59:52.379 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-10T19:59:52.379 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-10T19:59:53.705 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-10T19:59:53.786 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-10T19:59:56.734 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-10T19:59:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:00:00.985 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:00:01.407 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:00:03.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-10T20:00:03.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:00:03.968 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-04-10T20:00:04.210 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-10T20:00:04.210 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:00:04.210 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-10T20:00:04.267 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:00:04.267 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:00:04.267 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:00:04.267 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:00:05.098 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:00:05.268 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T20:00:05.269 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:00:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:00:05.407 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-04-10T20:00:05.445 DEBUG:teuthology.parallel:result is None 2024-04-10T20:00:13.891 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T20:00:13.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-10T20:00:13.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:00:28.801 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:00:28.824 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:00:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:00:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:00:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:00:29.713 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:00:29.713 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-10T20:00:29.714 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-10T20:00:29.714 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:00:29.714 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:00:29.714 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-10T20:00:29.714 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-10T20:00:29.851 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:00:30.133 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:00:30.133 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:00:30.133 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:00:30.134 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:00:30.134 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:00:30.134 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T20:00:30.134 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:00:30.134 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:00:30.356 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:00:30.356 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:00:30.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-04-10T20:00:30.391 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.02 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 129.el8 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Group : System Environment/Base 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 9265586 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Relocations : (not relocatable) 2024-04-10T20:00:30.392 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : CentOS BuildSystem 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-10T20:00:30.393 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:00:30.394 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:00:30.394 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:00:30.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:00:32.157 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-04-10T20:00:32.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-04-10T20:00:33.962 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-04-10T20:00:33.963 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-04-10T20:00:33.984 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.QZaX4avfEl 2024-04-10T20:00:33.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QZaX4avfEl 2024-04-10T20:00:34.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.QZaX4avfEl 2024-04-10T20:00:34.226 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.QZaX4avfEl is 6KB 2024-04-10T20:00:34.249 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.QZaX4avfEl 2024-04-10T20:00:34.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:00:34.339 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:00:34.340 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:00:34.340 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:00:34.340 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:00:34.341 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:00:34.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-04-10T20:00:41.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-10T20:00:41.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:01:04.500 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T20:01:04.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-10T20:01:04.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:01:22.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-04-10T20:01:31.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-10T20:01:31.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:01:41.320 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T20:01:53.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-10T20:01:53.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:01:53.692 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-04-10T20:01:54.043 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-10T20:01:54.043 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:01:54.044 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-04-10T20:01:54.062 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:01:54.063 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:01:54.063 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:01:54.063 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:01:55.063 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T20:01:55.064 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:01:55.139 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-04-10T20:01:55.151 DEBUG:teuthology.parallel:result is None 2024-04-10T20:02:32.006 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T20:02:44.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-10T20:02:44.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:02:44.356 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-04-10T20:02:44.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-10T20:02:44.641 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:02:44.642 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-04-10T20:02:44.662 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:02:44.662 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:02:44.662 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:02:44.662 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:02:45.663 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T20:02:45.664 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:02:45.742 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-04-10T20:02:45.760 DEBUG:teuthology.parallel:result is None 2024-04-10T20:02:45.761 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T20:02:45.768 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T20:02:45.768 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:02:45.771 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:02:45.774 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:02:45.796 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T20:02:45.803 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T20:02:45.813 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T20:02:45.813 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:02:45.832 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:02:45.836 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:02:45.886 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T20:02:45.892 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T20:02:45.893 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:02:45.902 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:02:45.918 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:02:45.945 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:02:45.954 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:02:45.957 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:02:45.966 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:02:45.968 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:02:45.977 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:02:45.978 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T20:02:45.985 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T20:02:45.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:02:45.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:02:46.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:02:46.039 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T20:02:46.048 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T20:02:46.048 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:02:46.069 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:02:46.071 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:02:46.098 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:02:46.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:02:46.233 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:02:46.332 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:02:46.363 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-10T20:02:46.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:02:46.434 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:02:46.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:02:46.551 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:02:46.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:02:46.641 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-10T20:02:46.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:02:46.709 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:02:46.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:02:46.812 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:02:46.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:02:46.943 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-10T20:02:46.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:02:47.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-04-10T20:02:47.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-04-10T20:02:47.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-04-10T20:02:47.081 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:02:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:02:47.124 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:02:47.508 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T20:02:47.515 INFO:teuthology.task.internal:Starting timer... 2024-04-10T20:02:47.515 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T20:02:47.525 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T20:02:47.533 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T20:02:47.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-04-10T20:02:47.599 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-04-10T20:02:47.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-04-10T20:02:47.682 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-04-10T20:02:47.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-04-10T20:02:47.755 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-04-10T20:02:47.757 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T20:02:47.757 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-04-10T20:02:47.782 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-04-10T20:02:47.783 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-04-10T20:02:47.830 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-04-10T20:02:47.831 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-04-10T20:02:47.891 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-04-10T20:02:47.892 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-04-10T20:02:47.892 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:02:47.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:02:47.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:02:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:02:47.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:02:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:02:47.978 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T20:02:47.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:02:48.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:02:48.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:02:48.081 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T20:02:48.093 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T20:02:48.098 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T20:02:48.315 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T20:02:48.330 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T20:02:48.331 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi121.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T20:20:58.144 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094/ansible_failures.yaml 2024-04-10T20:20:58.146 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T20:20:58.361 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d9d3f0efd1f544a4b40d4e87544f8167 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T20:20:58.363 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T20:20:58.374 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T20:20:58.374 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T20:20:58.388 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T20:20:58.474 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T20:20:58.541 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T20:20:58.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:20:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:20:58.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:20:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:20:58.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:20:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:20:58.703 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T20:20:58.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T20:20:58.740 INFO:teuthology.task.internal:Duration was 1091.225049 seconds 2024-04-10T20:20:58.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T20:20:58.756 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T20:20:58.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T20:20:58.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T20:20:58.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T20:20:58.808 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:20:58.810 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:20:58.811 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:20:59.079 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T20:20:59.080 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-04-10T20:20:59.080 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T20:20:59.107 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-04-10T20:20:59.107 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T20:20:59.132 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-04-10T20:20:59.133 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T20:20:59.155 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T20:20:59.155 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T20:20:59.158 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T20:20:59.175 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T20:20:59.236 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T20:20:59.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T20:20:59.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T20:20:59.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T20:20:59.350 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T20:20:59.360 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T20:20:59.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T20:20:59.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T20:20:59.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T20:20:59.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T20:20:59.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:20:59.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:20:59.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:20:59.502 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-04-10T20:20:59.502 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-04-10T20:20:59.508 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-04-10T20:20:59.535 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:20:59.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:20:59.593 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:20:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:20:59.610 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:20:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:20:59.626 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T20:20:59.638 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T20:20:59.639 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094/remote/smithi038 2024-04-10T20:20:59.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T20:20:59.751 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094/remote/smithi121 2024-04-10T20:20:59.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T20:20:59.809 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094/remote/smithi137 2024-04-10T20:20:59.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T20:20:59.868 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T20:20:59.868 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T20:20:59.870 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T20:20:59.872 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T20:20:59.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T20:20:59.903 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T20:20:59.903 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T20:20:59.918 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T20:20:59.918 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T20:20:59.930 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T20:20:59.932 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T20:20:59.947 INFO:teuthology.orchestra.run.smithi121.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 20:20 /home/ubuntu/cephtest 2024-04-10T20:20:59.948 INFO:teuthology.orchestra.run.smithi038.stdout: 525354 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 20:20 /home/ubuntu/cephtest 2024-04-10T20:20:59.949 INFO:teuthology.orchestra.run.smithi137.stdout: 525352 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 20:20 /home/ubuntu/cephtest 2024-04-10T20:20:59.951 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T20:20:59.962 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T20:20:59.992 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T20:21:00.027 DEBUG:teuthology.nuke:shortname: smithi038 2024-04-10T20:21:00.027 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:21:00.070 DEBUG:teuthology.nuke:shortname: smithi121 2024-04-10T20:21:00.071 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:21:00.099 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:39:33.488208', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJo7UEcUn7Zq95c1nc7PwdofWyPtmDUSPjfDf36LKarCV/ajFh7Z5IpYF1yYZe6EnwzdouwiRG4EL0jJfXtUv6o='} 2024-04-10T20:21:00.117 DEBUG:teuthology.nuke:shortname: smithi137 2024-04-10T20:21:00.117 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:21:00.133 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:39:33.489548', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIJzLfBkZvWg/mLkeY2xLDdUL71j6gEZjMB2CtN/KIWOwrzBHrNmq/zs0Ad5Bi2PddtglvcBKiDFGYkX6N37gdg='} 2024-04-10T20:21:00.191 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651094', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:39:33.486689', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZ5gdHLm+ka8Jj1+1GQObB3shjvBH94uu6BMKwdkH/xDz9wEOQLGYWeKuc4Bwbl7r7uG2NeqmhhglHiKDHIt5M='} 2024-04-10T20:21:00.197 INFO:teuthology.orchestra.console.smithi038:Power off 2024-04-10T20:21:00.198 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T20:21:00.238 INFO:teuthology.orchestra.console.smithi121:Power off 2024-04-10T20:21:00.239 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T20:21:00.253 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-04-10T20:21:00.253 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:00.278 INFO:teuthology.orchestra.console.smithi137:Power off 2024-04-10T20:21:00.278 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T20:21:00.287 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-04-10T20:21:00.288 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:00.301 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-04-10T20:21:00.331 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2024-04-10T20:21:00.331 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:00.340 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-10T20:21:00.382 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-04-10T20:21:04.303 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:04.341 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:04.383 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:04.415 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-04-10T20:21:04.451 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-10T20:21:04.492 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-04-10T20:21:08.420 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:08.453 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:08.493 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:08.531 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-04-10T20:21:08.562 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-10T20:21:08.602 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2024-04-10T20:21:08.632 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-04-10T20:21:08.703 INFO:teuthology.orchestra.console.smithi137:Power off completed 2024-04-10T20:21:12.564 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:21:12.673 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-04-10T20:21:12.774 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-04-10T20:21:12.880 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} duration: 1091.2250490188599 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d9d3f0efd1f544a4b40d4e87544f8167 status: dead success: false 2024-04-10T20:21:12.880 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T20:21:12.977 INFO:teuthology.run:DEAD