2024-04-10T16:54:21.285 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T16:54:21.285 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781 --verbose --owner scheduled_yuriw@teuthology --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} --name yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781/orig.config.yaml 2024-04-10T16:54:21.324 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T16:54:21.408 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781 branch: wip-yuri6-testing-2024-04-02-1310 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} email: yweinste@redhat.com first_in_suite: false job_id: '7650781' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'rbd object class functional tests ' name: yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 - volumes: size: 50 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2024-04-02-1310 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 filestore queue throttle high multiple: 2 filestore queue throttle max multiple: 10 journal throttle high multiple: 2 journal throttle max multiple: 10 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 9 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: a5074d4516d566e9d8b6aec912f26afd099de101 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a5074d4516d566e9d8b6aec912f26afd099de101 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-yuri6-testing-2024-04-02-1310 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 4372 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2024-04-02-1310 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a5074d4516d566e9d8b6aec912f26afd099de101 targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBATC+vDAb2Rcur+SDu3VeTKE/xE9QQakJ33eeBg5i0Lzhs1HCinlWnTSciE1slGZz0MSfEz/exvo24iBNblUea8= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIYq+vPtVYj9OOIODGaMk6Lb1wggSMJjEcDdjfyMx1JuMPuO1nEREONfVxjOG6zApWUGXECxjLay2do0kYwnYak= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHGyegUso0Nq+TgRfUXaZDOec33RbJBTHT9YqqQt8YvU5BhM2cIscSYGfdJ0hu5ooFCbneRiBqOgDUNxo3cs6A= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 3 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.2: - ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:17:51 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T16:54:21.408 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa; will attempt to use it 2024-04-10T16:54:21.409 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa/tasks 2024-04-10T16:54:21.409 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T16:54:21.411 INFO:teuthology.task.internal:Saving configuration 2024-04-10T16:54:21.424 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T16:54:21.430 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T16:54:21.455 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:47:26.798437', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBATC+vDAb2Rcur+SDu3VeTKE/xE9QQakJ33eeBg5i0Lzhs1HCinlWnTSciE1slGZz0MSfEz/exvo24iBNblUea8='} 2024-04-10T16:54:21.478 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:47:26.799837', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIYq+vPtVYj9OOIODGaMk6Lb1wggSMJjEcDdjfyMx1JuMPuO1nEREONfVxjOG6zApWUGXECxjLay2do0kYwnYak='} 2024-04-10T16:54:21.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:47:26.796614', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHGyegUso0Nq+TgRfUXaZDOec33RbJBTHT9YqqQt8YvU5BhM2cIscSYGfdJ0hu5ooFCbneRiBqOgDUNxo3cs6A='} 2024-04-10T16:54:21.500 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T16:54:21.505 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T16:54:21.506 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T16:54:21.506 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T16:54:21.506 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T16:54:21.609 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3f18edd160>, signals=[15]) 2024-04-10T16:54:21.609 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T16:54:21.615 INFO:teuthology.task.internal:Opening connections... 2024-04-10T16:54:21.615 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-04-10T16:54:21.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:54:21.690 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-04-10T16:54:21.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:54:21.762 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-04-10T16:54:21.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:54:21.834 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T16:54:21.840 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-04-10T16:54:21.863 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-04-10T16:54:21.863 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-04-10T16:54:21.923 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-04-10T16:54:21.923 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="8" 2024-04-10T16:54:21.923 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-04-10T16:54:21.923 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-04-10T16:54:21.923 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="8" 2024-04-10T16:54:21.924 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el8" 2024-04-10T16:54:21.924 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T16:54:21.924 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-04-10T16:54:21.924 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T16:54:21.924 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-04-10T16:54:21.924 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T16:54:21.924 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T16:54:21.924 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T16:54:21.924 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-04-10T16:54:21.950 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-04-10T16:54:21.971 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-04-10T16:54:21.971 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-04-10T16:54:22.033 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-04-10T16:54:22.033 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8" 2024-04-10T16:54:22.033 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-04-10T16:54:22.033 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-04-10T16:54:22.034 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8" 2024-04-10T16:54:22.034 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8" 2024-04-10T16:54:22.034 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T16:54:22.034 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-04-10T16:54:22.034 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T16:54:22.034 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-04-10T16:54:22.034 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T16:54:22.034 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T16:54:22.034 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T16:54:22.034 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-04-10T16:54:22.057 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-04-10T16:54:22.079 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-04-10T16:54:22.079 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-04-10T16:54:22.140 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-04-10T16:54:22.140 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8" 2024-04-10T16:54:22.140 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-04-10T16:54:22.140 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-04-10T16:54:22.141 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8" 2024-04-10T16:54:22.141 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2024-04-10T16:54:22.141 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T16:54:22.141 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-04-10T16:54:22.141 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T16:54:22.141 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-04-10T16:54:22.141 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T16:54:22.141 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T16:54:22.141 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T16:54:22.141 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-04-10T16:54:22.157 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T16:54:22.166 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T16:54:22.177 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T16:54:22.177 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-04-10T16:54:22.180 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-04-10T16:54:22.182 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-04-10T16:54:22.201 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T16:54:22.207 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T16:54:22.207 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-04-10T16:54:22.241 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-04-10T16:54:22.243 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-04-10T16:54:22.285 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T16:54:22.362 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T16:54:22.376 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T16:54:22.377 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T16:54:22.377 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T16:54:22.377 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T16:54:22.378 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T16:54:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:54:22.398 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-10T16:54:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:54:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:54:22.399 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-04-10T16:54:22.400 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-04-10T16:54:22.459 INFO:teuthology.orchestra.run.smithi070.stdout:4.18.0-489.el8.x86_64 2024-04-10T16:54:22.459 INFO:teuthology.task.kernel:Running kernel on smithi070: 4.18.0-489.el8.x86_64 2024-04-10T16:54:22.460 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-04-10T16:54:22.461 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-489.el8.x86_64 2024-04-10T16:54:22.461 INFO:teuthology.task.kernel:Running kernel on smithi163: 4.18.0-489.el8.x86_64 2024-04-10T16:54:22.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2024-04-10T16:54:22.462 INFO:teuthology.orchestra.run.smithi146.stdout:4.18.0-489.el8.x86_64 2024-04-10T16:54:22.462 INFO:teuthology.task.kernel:Running kernel on smithi146: 4.18.0-489.el8.x86_64 2024-04-10T16:54:22.462 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-04-10T16:54:28.093 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - AppStream 8.8 MB/s | 28 MB 00:03 2024-04-10T16:54:28.368 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - AppStream 8.4 MB/s | 28 MB 00:03 2024-04-10T16:54:28.368 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - AppStream 8.6 MB/s | 28 MB 00:03 2024-04-10T16:54:35.577 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - BaseOS 4.3 MB/s | 10 MB 00:02 2024-04-10T16:56:46.358 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - BaseOS 79 kB/s | 10 MB 02:12 2024-04-10T16:56:47.501 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - Extras 140 B/s | 18 kB 02:10 2024-04-10T16:56:48.974 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - Extras 17 kB/s | 18 kB 00:01 2024-04-10T16:56:49.240 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - BaseOS 78 kB/s | 10 MB 02:15 2024-04-10T16:59:03.204 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras 138 B/s | 18 kB 02:12 2024-04-10T16:59:42.272 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - Extras common packages 45 B/s | 7.7 kB 02:54 2024-04-10T16:59:42.282 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - Extras common packages 45 B/s | 7.7 kB 02:53 2024-04-10T17:01:10.713 INFO:teuthology.orchestra.run.smithi146.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T17:01:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T17:01:10.714 INFO:teuthology.orchestra.run.smithi146.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T17:01:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T17:01:10.718 INFO:teuthology.orchestra.run.smithi070.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T17:01:10.719 INFO:teuthology.orchestra.run.smithi070.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T17:01:10.719 INFO:teuthology.orchestra.run.smithi070.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T17:01:10.719 INFO:teuthology.orchestra.run.smithi070.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T17:02:00.170 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras common packages 45 B/s | 7.7 kB 02:56 2024-04-10T17:02:16.522 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 253 kB/s | 16 MB 01:05 2024-04-10T17:02:20.261 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 377 kB/s | 24 kB 00:00 2024-04-10T17:02:20.402 INFO:teuthology.orchestra.run.smithi070.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T17:02:23.236 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T17:02:23.236 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T17:02:23.364 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-10T17:02:23.365 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-10T17:02:23.366 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-10T17:02:23.366 INFO:teuthology.orchestra.run.smithi070.stdout:Install 3 Packages 2024-04-10T17:02:23.366 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-10T17:02:23.366 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 90 M 2024-04-10T17:02:23.366 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 96 M 2024-04-10T17:02:23.366 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-10T17:02:26.137 INFO:teuthology.orchestra.run.smithi070.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 4.3 MB/s | 10 MB 00:02 2024-04-10T17:02:28.936 INFO:teuthology.orchestra.run.smithi070.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 7.0 MB/s | 36 MB 00:05 2024-04-10T17:02:29.645 INFO:teuthology.orchestra.run.smithi070.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 7.4 MB/s | 44 MB 00:05 2024-04-10T17:02:29.645 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-10T17:02:29.645 INFO:teuthology.orchestra.run.smithi070.stdout:Total 14 MB/s | 90 MB 00:06 2024-04-10T17:02:30.327 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-10T17:02:30.741 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-10T17:02:30.742 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-10T17:02:31.988 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-10T17:02:32.021 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-10T17:02:34.827 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-10T17:02:35.360 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:02:38.977 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:02:39.367 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:02:42.802 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:02:43.099 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:03:05.064 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:03:05.088 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:03:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:03:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:03:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:03:06.086 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:03:06.086 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-10T17:03:06.087 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-10T17:03:06.087 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T17:03:06.087 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T17:03:06.087 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-10T17:03:06.087 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-10T17:03:06.241 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:03:06.458 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:03:06.458 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:03:06.458 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T17:03:06.458 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T17:03:06.458 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T17:03:06.459 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T17:03:06.459 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T17:03:06.459 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:03:06.669 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:03:06.669 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:03:06.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2024-04-10T17:03:06.702 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.02 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 129.el8 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Group : System Environment/Base 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 9265586 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Relocations : (not relocatable) 2024-04-10T17:03:06.703 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : CentOS BuildSystem 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-10T17:03:06.704 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T17:03:06.705 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T17:03:06.705 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 4.18.0-552.el8.x86_64 2024-04-10T17:03:06.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T17:03:08.194 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-04-10T17:03:09.996 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-04-10T17:03:09.998 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-04-10T17:03:10.029 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.7VSnYvl4c4 2024-04-10T17:03:10.030 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7VSnYvl4c4 2024-04-10T17:03:10.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.7VSnYvl4c4 2024-04-10T17:03:10.248 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.7VSnYvl4c4 is 6KB 2024-04-10T17:03:10.295 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.7VSnYvl4c4 2024-04-10T17:03:10.313 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T17:03:10.382 INFO:teuthology.orchestra.run.smithi070.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T17:03:10.382 INFO:teuthology.orchestra.run.smithi070.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T17:03:10.383 INFO:teuthology.orchestra.run.smithi070.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T17:03:10.383 INFO:teuthology.orchestra.run.smithi070.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T17:03:10.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T17:03:10.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-04-10T17:03:28.617 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T17:03:28.617 INFO:teuthology.orchestra.run.smithi163.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T17:03:28.617 INFO:teuthology.orchestra.run.smithi163.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T17:03:28.617 INFO:teuthology.orchestra.run.smithi163.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T17:03:30.861 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 16 MB 00:02 2024-04-10T17:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 407 kB/s | 24 kB 00:00 2024-04-10T17:03:34.784 INFO:teuthology.orchestra.run.smithi163.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T17:03:37.548 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T17:03:37.548 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T17:03:37.670 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-04-10T17:03:37.671 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-04-10T17:03:37.672 INFO:teuthology.orchestra.run.smithi163.stdout:Install 3 Packages 2024-04-10T17:03:37.672 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-10T17:03:37.672 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 90 M 2024-04-10T17:03:37.672 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 96 M 2024-04-10T17:03:37.672 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-04-10T17:03:40.563 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T17:03:40.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-10T17:03:40.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:03:59.010 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-10T17:04:08.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-10T17:04:08.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:04:11.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-10T17:04:23.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-10T17:04:23.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:04:26.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-10T17:04:41.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-10T17:04:41.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:04:44.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-10T17:04:47.516 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 77 kB/s | 16 MB 03:36 2024-04-10T17:04:51.273 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 463 kB/s | 24 kB 00:00 2024-04-10T17:04:51.416 INFO:teuthology.orchestra.run.smithi146.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T17:04:54.228 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T17:04:54.228 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T17:04:54.338 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-04-10T17:04:54.338 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout:Install 3 Packages 2024-04-10T17:04:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-10T17:04:54.340 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 90 M 2024-04-10T17:04:54.340 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 96 M 2024-04-10T17:04:54.340 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-04-10T17:05:02.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-10T17:05:02.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:05:02.619 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-10T17:05:02.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-10T17:05:02.886 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T17:05:02.886 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-10T17:05:02.946 INFO:teuthology.orchestra.run.smithi070.stdout:4.18.0-552.el8.x86_64 2024-04-10T17:05:02.946 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T17:05:02.946 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T17:05:02.946 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T17:05:03.947 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T17:05:03.947 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T17:05:04.022 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-04-10T17:05:04.035 DEBUG:teuthology.parallel:result is None 2024-04-10T17:05:52.640 INFO:teuthology.orchestra.run.smithi163.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 4.1 MB/s | 10 MB 00:02 2024-04-10T17:05:54.198 INFO:teuthology.orchestra.run.smithi163.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 9.0 MB/s | 36 MB 00:04 2024-04-10T17:05:54.815 INFO:teuthology.orchestra.run.smithi163.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 9.3 MB/s | 44 MB 00:04 2024-04-10T17:05:54.815 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-04-10T17:05:54.815 INFO:teuthology.orchestra.run.smithi163.stdout:Total 675 kB/s | 90 MB 02:17 2024-04-10T17:05:55.498 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-04-10T17:05:55.927 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-04-10T17:05:55.927 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-04-10T17:05:57.164 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-04-10T17:05:57.196 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-04-10T17:06:00.011 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-04-10T17:06:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:06:04.197 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:06:04.605 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:06:08.049 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:06:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:06:30.661 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:06:30.685 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:06:31.037 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:06:31.037 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:06:31.037 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:06:31.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:06:31.465 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-10T17:06:31.465 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-04-10T17:06:31.465 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T17:06:31.465 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T17:06:31.465 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-10T17:06:31.465 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-04-10T17:06:31.601 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:06:31.860 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:06:31.860 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:06:31.860 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T17:06:31.860 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T17:06:31.860 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T17:06:31.860 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T17:06:31.861 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T17:06:31.861 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:06:32.071 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:06:32.072 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:06:32.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2024-04-10T17:06:32.105 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2024-04-10T17:06:32.105 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2024-04-10T17:06:32.105 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.02 2024-04-10T17:06:32.105 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 129.el8 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Group : System Environment/Base 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 9265586 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Relocations : (not relocatable) 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : CentOS BuildSystem 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2024-04-10T17:06:32.106 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2024-04-10T17:06:32.107 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-10T17:06:32.107 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T17:06:32.107 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T17:06:32.107 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T17:06:32.107 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2024-04-10T17:06:32.107 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-04-10T17:06:32.107 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T17:06:32.108 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T17:06:32.108 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 4.18.0-552.el8.x86_64 2024-04-10T17:06:32.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T17:06:33.587 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2024-04-10T17:06:35.807 INFO:teuthology.orchestra.run.smithi163.stderr:done 2024-04-10T17:06:35.809 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-04-10T17:06:35.850 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.xdXa9EJ8Pk 2024-04-10T17:06:35.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xdXa9EJ8Pk 2024-04-10T17:06:35.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.xdXa9EJ8Pk 2024-04-10T17:06:36.079 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.xdXa9EJ8Pk is 6KB 2024-04-10T17:06:36.127 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.xdXa9EJ8Pk 2024-04-10T17:06:36.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T17:06:36.218 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T17:06:36.218 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T17:06:36.218 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T17:06:36.218 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T17:06:36.219 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T17:06:36.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2024-04-10T17:07:06.418 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T17:07:06.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-10T17:07:06.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:07:12.738 INFO:teuthology.orchestra.run.smithi146.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 1.4 MB/s | 10 MB 00:07 2024-04-10T17:07:16.455 INFO:teuthology.orchestra.run.smithi146.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 4.0 MB/s | 44 MB 00:10 2024-04-10T17:07:16.813 INFO:teuthology.orchestra.run.smithi146.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 3.2 MB/s | 36 MB 00:11 2024-04-10T17:07:16.814 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-04-10T17:07:16.814 INFO:teuthology.orchestra.run.smithi146.stdout:Total 650 kB/s | 90 MB 02:22 2024-04-10T17:07:17.501 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-04-10T17:07:17.976 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-04-10T17:07:17.976 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-04-10T17:07:19.241 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-04-10T17:07:19.278 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-04-10T17:07:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-04-10T17:07:22.706 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:07:26.336 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:07:26.889 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:07:30.343 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:07:30.656 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:07:40.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-04-10T17:07:49.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-10T17:07:49.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:07:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:07:52.892 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:07:53.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-04-10T17:07:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:07:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:07:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:07:53.762 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:07:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-10T17:07:53.763 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-04-10T17:07:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T17:07:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T17:07:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-10T17:07:53.763 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-04-10T17:07:53.919 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:07:54.143 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:07:54.143 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:07:54.143 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T17:07:54.143 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T17:07:54.143 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T17:07:54.143 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T17:07:54.143 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T17:07:54.144 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:07:54.356 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:07:54.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:07:54.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.02 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 129.el8 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:Group : System Environment/Base 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 9265586 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-04-10T17:07:54.391 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:Relocations : (not relocatable) 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : CentOS BuildSystem 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-10T17:07:54.392 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T17:07:54.393 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T17:07:54.393 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 4.18.0-552.el8.x86_64 2024-04-10T17:07:54.393 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T17:07:55.866 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-04-10T17:07:57.789 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-04-10T17:07:57.791 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-04-10T17:07:57.817 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.VBGGhVa4p1 2024-04-10T17:07:57.817 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VBGGhVa4p1 2024-04-10T17:07:57.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.VBGGhVa4p1 2024-04-10T17:07:58.034 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.VBGGhVa4p1 is 6KB 2024-04-10T17:07:58.082 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.VBGGhVa4p1 2024-04-10T17:07:58.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T17:07:58.188 INFO:teuthology.orchestra.run.smithi146.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T17:07:58.188 INFO:teuthology.orchestra.run.smithi146.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T17:07:58.188 INFO:teuthology.orchestra.run.smithi146.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T17:07:58.188 INFO:teuthology.orchestra.run.smithi146.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T17:07:58.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T17:07:58.298 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-04-10T17:08:05.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-10T17:08:05.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:08:28.300 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T17:08:28.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-10T17:08:28.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:09:02.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-04-10T17:09:05.025 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T17:09:11.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-10T17:09:11.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:09:20.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-10T17:09:20.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:09:20.427 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-04-10T17:09:20.730 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-04-10T17:09:20.730 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T17:09:20.730 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-04-10T17:09:20.749 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-552.el8.x86_64 2024-04-10T17:09:20.750 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T17:09:20.750 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T17:09:20.750 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T17:09:21.751 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T17:09:21.751 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T17:09:21.827 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2024-04-10T17:09:21.839 DEBUG:teuthology.parallel:result is None 2024-04-10T17:09:30.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-04-10T17:09:42.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-10T17:09:42.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:09:42.651 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-04-10T17:09:42.933 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-10T17:09:42.933 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T17:09:42.933 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-04-10T17:09:42.991 INFO:teuthology.orchestra.run.smithi146.stdout:4.18.0-552.el8.x86_64 2024-04-10T17:09:42.991 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T17:09:42.991 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T17:09:42.991 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T17:09:43.992 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T17:09:43.993 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T17:09:44.093 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-04-10T17:09:44.105 DEBUG:teuthology.parallel:result is None 2024-04-10T17:09:44.105 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T17:09:44.116 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T17:09:44.117 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T17:09:44.120 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T17:09:44.122 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T17:09:44.145 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T17:09:44.152 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T17:09:44.161 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T17:09:44.161 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T17:09:44.182 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T17:09:44.184 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T17:09:44.250 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T17:09:44.256 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T17:09:44.256 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T17:09:44.258 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T17:09:44.261 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T17:09:44.291 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:09:44.301 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:09:44.302 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:09:44.312 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:09:44.321 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:09:44.330 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:09:44.332 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T17:09:44.338 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T17:09:44.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T17:09:44.346 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T17:09:44.348 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T17:09:44.385 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T17:09:44.402 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T17:09:44.402 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T17:09:44.416 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T17:09:44.418 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T17:09:44.444 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:09:44.520 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:09:44.569 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:09:44.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:09:44.696 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-10T17:09:44.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T17:09:44.763 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:09:44.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:09:44.865 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:09:44.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:09:44.993 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-10T17:09:44.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T17:09:45.062 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:09:45.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:09:45.165 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:09:45.245 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:09:45.297 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-04-10T17:09:45.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T17:09:45.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-04-10T17:09:45.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-04-10T17:09:45.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-04-10T17:09:45.429 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:09:45.437 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:09:45.472 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:09:45.907 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T17:09:45.915 INFO:teuthology.task.internal:Starting timer... 2024-04-10T17:09:45.915 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T17:09:45.924 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T17:09:45.932 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T17:09:45.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-04-10T17:09:46.002 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-04-10T17:09:46.004 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-04-10T17:09:46.079 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-04-10T17:09:46.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-04-10T17:09:46.161 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-04-10T17:09:46.163 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T17:09:46.163 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-04-10T17:09:46.234 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-04-10T17:09:46.235 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-04-10T17:09:46.263 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-04-10T17:09:46.263 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-04-10T17:09:46.291 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-04-10T17:09:46.291 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2024-04-10T17:09:46.291 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:09:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:09:46.318 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:09:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:09:46.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:09:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:09:46.373 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T17:09:46.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-04-10T17:09:46.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-04-10T17:09:46.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-04-10T17:09:46.452 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T17:09:46.462 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T17:09:46.467 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T17:09:46.614 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T17:09:46.629 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T17:09:46.630 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com,smithi146.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T17:28:03.343 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781/ansible_failures.yaml 2024-04-10T17:28:03.345 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T17:28:03.569 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=86efc5142eb4401492c00391f43ea0bc Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T17:28:03.571 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T17:28:03.581 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T17:28:03.581 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T17:28:03.647 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T17:28:03.714 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T17:28:03.777 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T17:28:03.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:28:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:28:03.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:28:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:28:03.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:28:03.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:28:03.940 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T17:28:03.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T17:28:03.960 INFO:teuthology.task.internal:Duration was 1098.045305 seconds 2024-04-10T17:28:03.960 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T17:28:03.969 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T17:28:03.969 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T17:28:03.971 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T17:28:03.973 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T17:28:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:28:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:28:04.027 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:28:04.475 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T17:28:04.476 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-04-10T17:28:04.477 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T17:28:04.503 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-04-10T17:28:04.504 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T17:28:04.531 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-04-10T17:28:04.531 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T17:28:04.554 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T17:28:04.554 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T17:28:04.557 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T17:28:04.574 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T17:28:04.635 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T17:28:04.635 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T17:28:04.638 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T17:28:04.640 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T17:28:04.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T17:28:04.756 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T17:28:04.756 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T17:28:04.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T17:28:04.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T17:28:04.836 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T17:28:04.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:28:04.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:28:04.852 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:28:04.871 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-04-10T17:28:04.876 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-04-10T17:28:04.902 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-04-10T17:28:04.925 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:28:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:28:04.959 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:28:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:28:04.976 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:28:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:28:04.992 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T17:28:05.001 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T17:28:05.002 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781/remote/smithi070 2024-04-10T17:28:05.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T17:28:05.071 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781/remote/smithi146 2024-04-10T17:28:05.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T17:28:05.127 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781/remote/smithi163 2024-04-10T17:28:05.127 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T17:28:05.186 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T17:28:05.186 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T17:28:05.188 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T17:28:05.190 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T17:28:05.208 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T17:28:05.218 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T17:28:05.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T17:28:05.226 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T17:28:05.226 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T17:28:05.246 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T17:28:05.248 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T17:28:05.261 INFO:teuthology.orchestra.run.smithi070.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 17:28 /home/ubuntu/cephtest 2024-04-10T17:28:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: 525553 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 17:28 /home/ubuntu/cephtest 2024-04-10T17:28:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 17:28 /home/ubuntu/cephtest 2024-04-10T17:28:05.267 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T17:28:05.275 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T17:28:05.292 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T17:28:05.310 DEBUG:teuthology.nuke:shortname: smithi070 2024-04-10T17:28:05.311 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T17:28:05.332 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:47:26.798437', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBATC+vDAb2Rcur+SDu3VeTKE/xE9QQakJ33eeBg5i0Lzhs1HCinlWnTSciE1slGZz0MSfEz/exvo24iBNblUea8='} 2024-04-10T17:28:05.336 DEBUG:teuthology.nuke:shortname: smithi146 2024-04-10T17:28:05.336 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T17:28:05.356 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:47:26.799837', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIYq+vPtVYj9OOIODGaMk6Lb1wggSMJjEcDdjfyMx1JuMPuO1nEREONfVxjOG6zApWUGXECxjLay2do0kYwnYak='} 2024-04-10T17:28:05.361 DEBUG:teuthology.nuke:shortname: smithi163 2024-04-10T17:28:05.361 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T17:28:05.385 INFO:teuthology.orchestra.console.smithi070:Power off 2024-04-10T17:28:05.386 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T17:28:05.397 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:47:26.796614', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHGyegUso0Nq+TgRfUXaZDOec33RbJBTHT9YqqQt8YvU5BhM2cIscSYGfdJ0hu5ooFCbneRiBqOgDUNxo3cs6A='} 2024-04-10T17:28:05.410 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2024-04-10T17:28:05.410 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:05.426 INFO:teuthology.orchestra.console.smithi146:Power off 2024-04-10T17:28:05.427 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T17:28:05.437 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-10T17:28:05.448 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-04-10T17:28:05.448 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:05.461 INFO:teuthology.orchestra.console.smithi163:Power off 2024-04-10T17:28:05.462 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T17:28:05.470 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-04-10T17:28:05.483 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-04-10T17:28:05.483 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:05.505 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-04-10T17:28:09.438 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:09.472 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:09.506 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:09.548 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-10T17:28:09.581 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-04-10T17:28:09.615 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-04-10T17:28:13.550 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:13.583 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:13.616 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:13.660 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2024-04-10T17:28:13.691 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-04-10T17:28:13.724 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-04-10T17:28:13.761 INFO:teuthology.orchestra.console.smithi070:Power off completed 2024-04-10T17:28:13.792 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-04-10T17:28:17.726 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:28:17.837 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-04-10T17:28:17.938 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-04-10T17:28:18.045 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} duration: 1098.0453045368195 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=86efc5142eb4401492c00391f43ea0bc status: dead success: false 2024-04-10T17:28:18.046 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T17:28:18.138 INFO:teuthology.run:DEAD