2024-04-10T20:33:23.536 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T20:33:23.537 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} --archive /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203 --name yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203/orig.config.yaml 2024-04-10T20:33:23.579 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T20:33:23.674 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203 branch: wip-yuri10-testing-2024-03-24-1159 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} email: yweinste@redhat.com first_in_suite: false job_id: '7651203' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-03-24-1159 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - must scrub before tier agent can activate sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri10-testing-2024-03-24-1159 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 8148 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-03-24-1159 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJt6LKF7+imv80NqA/P4Msf0C5EwDQIqLwTQTEIlwH+kqnllR+q9GBuvrnPkwdfUEEMl1zRK9WvEwWhGBhjO0c= smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwnOZA3Tg0HffDpwLQTQqn3CyIMvS/EOob7YyfWGnybPWQM9sS6rozkWndBJS9IgYaOOSO+1ks6tMM2b4HBc+o= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+tbx6m0sf6E0EWToc5AFdYRYf6GMe6grZUxN4XzMSKVyaZhjKz3yDh+q/RC/JzzjOXzBc+GLXGRL0ZebyyQ0I= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: pacific exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 3600 - sudo ceph osd pool set cache target_max_objects 250 - sudo ceph osd pool set cache min_read_recency_for_promote 2 - rados: clients: - client.2 objects: 500 op_weights: cache_evict: 50 cache_flush: 50 cache_try_flush: 50 copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:20:47 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T20:33:23.674 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa; will attempt to use it 2024-04-10T20:33:23.675 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks 2024-04-10T20:33:23.675 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T20:33:23.676 INFO:teuthology.task.internal:Saving configuration 2024-04-10T20:33:23.691 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T20:33:23.699 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:33:23.733 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:26:36.006686', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJt6LKF7+imv80NqA/P4Msf0C5EwDQIqLwTQTEIlwH+kqnllR+q9GBuvrnPkwdfUEEMl1zRK9WvEwWhGBhjO0c='} 2024-04-10T20:33:23.767 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:26:36.004711', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwnOZA3Tg0HffDpwLQTQqn3CyIMvS/EOob7YyfWGnybPWQM9sS6rozkWndBJS9IgYaOOSO+1ks6tMM2b4HBc+o='} 2024-04-10T20:33:23.804 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:26:36.008226', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+tbx6m0sf6E0EWToc5AFdYRYf6GMe6grZUxN4XzMSKVyaZhjKz3yDh+q/RC/JzzjOXzBc+GLXGRL0ZebyyQ0I='} 2024-04-10T20:33:23.804 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T20:33:23.812 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T20:33:23.812 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T20:33:23.812 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T20:33:23.812 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T20:33:23.972 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f65191d7160>, signals=[15]) 2024-04-10T20:33:23.972 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T20:33:23.978 INFO:teuthology.task.internal:Opening connections... 2024-04-10T20:33:23.979 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-04-10T20:33:23.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:33:24.079 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-04-10T20:33:24.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:33:24.163 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-04-10T20:33:24.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:33:24.242 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T20:33:24.248 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-04-10T20:33:24.271 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-04-10T20:33:24.271 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-04-10T20:33:24.331 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-04-10T20:33:24.331 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8" 2024-04-10T20:33:24.331 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-04-10T20:33:24.331 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-04-10T20:33:24.331 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8" 2024-04-10T20:33:24.331 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8" 2024-04-10T20:33:24.331 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T20:33:24.331 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-04-10T20:33:24.331 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T20:33:24.332 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-04-10T20:33:24.332 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T20:33:24.332 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T20:33:24.332 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T20:33:24.332 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-04-10T20:33:24.365 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-04-10T20:33:24.388 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-04-10T20:33:24.388 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-04-10T20:33:24.448 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2024-04-10T20:33:24.448 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="8" 2024-04-10T20:33:24.448 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="8" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el8" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T20:33:24.449 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T20:33:24.449 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-04-10T20:33:24.481 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-04-10T20:33:24.503 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-04-10T20:33:24.503 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-04-10T20:33:24.564 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2024-04-10T20:33:24.564 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8" 2024-04-10T20:33:24.564 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T20:33:24.565 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T20:33:24.565 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-04-10T20:33:24.598 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T20:33:24.607 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T20:33:24.643 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T20:33:24.643 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-04-10T20:33:24.646 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-04-10T20:33:24.648 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-04-10T20:33:24.667 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T20:33:24.675 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T20:33:24.675 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-04-10T20:33:24.706 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-04-10T20:33:24.709 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-04-10T20:33:24.734 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T20:33:24.842 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T20:33:24.857 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T20:33:24.857 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T20:33:24.858 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T20:33:24.858 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T20:33:24.859 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T20:33:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:33:24.879 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-04-10T20:33:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:33:24.880 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-04-10T20:33:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:33:24.881 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-04-10T20:33:24.938 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-489.el8.x86_64 2024-04-10T20:33:24.938 INFO:teuthology.task.kernel:Running kernel on smithi187: 4.18.0-489.el8.x86_64 2024-04-10T20:33:24.938 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2024-04-10T20:33:24.942 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-489.el8.x86_64 2024-04-10T20:33:24.942 INFO:teuthology.task.kernel:Running kernel on smithi088: 4.18.0-489.el8.x86_64 2024-04-10T20:33:24.942 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-04-10T20:33:24.943 INFO:teuthology.orchestra.run.smithi161.stdout:4.18.0-489.el8.x86_64 2024-04-10T20:33:24.943 INFO:teuthology.task.kernel:Running kernel on smithi161: 4.18.0-489.el8.x86_64 2024-04-10T20:33:24.943 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2024-04-10T20:33:28.726 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - AppStream 23 MB/s | 28 MB 00:01 2024-04-10T20:33:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - BaseOS 2.3 MB/s | 10 MB 00:04 2024-04-10T20:34:24.543 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras 409 B/s | 18 kB 00:44 2024-04-10T20:35:37.913 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 8 - AppStream 219 kB/s | 28 MB 02:10 2024-04-10T20:35:38.221 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 8 - AppStream 219 kB/s | 28 MB 02:10 2024-04-10T20:35:45.678 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 8 - BaseOS 3.8 MB/s | 10 MB 00:02 2024-04-10T20:35:45.920 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 8 - BaseOS 3.8 MB/s | 10 MB 00:02 2024-04-10T20:36:31.641 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 8 - Extras 413 B/s | 18 kB 00:44 2024-04-10T20:36:31.821 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 8 - Extras 409 B/s | 18 kB 00:44 2024-04-10T20:36:36.226 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras common packages 60 B/s | 7.7 kB 02:11 2024-04-10T20:38:04.673 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T20:38:04.674 INFO:teuthology.orchestra.run.smithi088.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T20:38:04.674 INFO:teuthology.orchestra.run.smithi088.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T20:38:04.674 INFO:teuthology.orchestra.run.smithi088.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T20:38:05.393 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2024-04-10T20:38:09.126 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 574 kB/s | 24 kB 00:00 2024-04-10T20:38:09.267 INFO:teuthology.orchestra.run.smithi088.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T20:38:12.090 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T20:38:12.090 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T20:38:12.219 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-04-10T20:38:12.220 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-04-10T20:38:12.220 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-04-10T20:38:12.220 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-04-10T20:38:12.220 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-04-10T20:38:12.220 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T20:38:12.220 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-04-10T20:38:12.220 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T20:38:12.221 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T20:38:12.221 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T20:38:12.221 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-04-10T20:38:12.221 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-04-10T20:38:12.221 INFO:teuthology.orchestra.run.smithi088.stdout:Install 3 Packages 2024-04-10T20:38:12.221 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T20:38:12.222 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 90 M 2024-04-10T20:38:12.222 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 96 M 2024-04-10T20:38:12.222 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-04-10T20:38:41.327 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 8 - Extras common packages 61 B/s | 7.7 kB 02:09 2024-04-10T20:38:41.883 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 8 - Extras common packages 61 B/s | 7.7 kB 02:10 2024-04-10T20:40:09.728 INFO:teuthology.orchestra.run.smithi161.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:27 2024-04-10T20:40:09.729 INFO:teuthology.orchestra.run.smithi161.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T20:40:09.729 INFO:teuthology.orchestra.run.smithi161.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T20:40:09.730 INFO:teuthology.orchestra.run.smithi161.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T20:40:09.767 INFO:teuthology.orchestra.run.smithi187.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T20:40:09.767 INFO:teuthology.orchestra.run.smithi187.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T20:40:09.767 INFO:teuthology.orchestra.run.smithi187.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T20:40:09.767 INFO:teuthology.orchestra.run.smithi187.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T20:40:11.500 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 16 MB 00:01 2024-04-10T20:40:15.305 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 421 kB/s | 24 kB 00:00 2024-04-10T20:40:15.446 INFO:teuthology.orchestra.run.smithi187.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T20:40:18.310 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T20:40:18.311 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T20:40:18.422 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-10T20:40:18.422 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout:Install 3 Packages 2024-04-10T20:40:18.423 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-10T20:40:18.424 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 90 M 2024-04-10T20:40:18.424 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 96 M 2024-04-10T20:40:18.424 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-10T20:40:23.226 INFO:teuthology.orchestra.run.smithi088.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 7.4 MB/s | 10 MB 00:01 2024-04-10T20:40:41.498 INFO:teuthology.orchestra.run.smithi088.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 1.9 MB/s | 36 MB 00:19 2024-04-10T20:40:51.138 INFO:teuthology.orchestra.run.smithi088.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 1.5 MB/s | 44 MB 00:29 2024-04-10T20:40:51.138 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-04-10T20:40:51.138 INFO:teuthology.orchestra.run.smithi088.stdout:Total 583 kB/s | 90 MB 02:38 2024-04-10T20:40:51.822 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-04-10T20:40:52.265 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-04-10T20:40:52.266 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-04-10T20:40:53.494 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-04-10T20:40:53.519 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-04-10T20:40:56.303 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-04-10T20:40:56.826 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:41:00.387 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:41:00.750 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:41:04.185 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:41:04.456 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:41:26.295 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:41:26.318 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:41:26.658 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:41:26.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:41:26.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:41:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:41:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T20:41:27.141 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-04-10T20:41:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:41:27.141 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:41:27.142 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T20:41:27.142 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-04-10T20:41:27.288 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:41:27.568 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:41:27.568 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:41:27.569 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:41:27.569 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:41:27.569 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:41:27.569 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T20:41:27.569 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:41:27.569 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:41:27.780 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:41:27.780 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:41:27.780 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.02 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 129.el8 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Group : System Environment/Base 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 9265586 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:41:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:Relocations : (not relocatable) 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : CentOS BuildSystem 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T20:41:27.814 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:41:27.815 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:41:27.815 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:41:27.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:41:29.029 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-04-10T20:41:31.281 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-04-10T20:41:31.283 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-04-10T20:41:31.311 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.S41VAH2zOG 2024-04-10T20:41:31.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.S41VAH2zOG 2024-04-10T20:41:31.385 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.S41VAH2zOG 2024-04-10T20:41:31.532 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.S41VAH2zOG is 6KB 2024-04-10T20:41:31.553 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.S41VAH2zOG 2024-04-10T20:41:31.570 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:41:31.645 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:41:31.645 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:41:31.645 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:41:31.645 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:41:31.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:41:31.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-04-10T20:42:01.805 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T20:42:01.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T20:42:01.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:42:20.319 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-10T20:42:29.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T20:42:29.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:42:32.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-10T20:42:34.627 INFO:teuthology.orchestra.run.smithi187.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 2.5 MB/s | 10 MB 00:04 2024-04-10T20:42:44.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T20:42:44.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:42:47.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-10T20:42:48.625 INFO:teuthology.orchestra.run.smithi187.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 2.0 MB/s | 36 MB 00:18 2024-04-10T20:42:56.307 INFO:teuthology.orchestra.run.smithi187.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 1.7 MB/s | 44 MB 00:25 2024-04-10T20:42:56.307 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-10T20:42:56.307 INFO:teuthology.orchestra.run.smithi187.stdout:Total 587 kB/s | 90 MB 02:37 2024-04-10T20:42:56.993 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-10T20:42:57.426 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-10T20:42:57.427 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-10T20:42:58.682 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-10T20:42:58.703 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-10T20:43:01.538 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-10T20:43:02.141 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:43:02.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T20:43:02.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:43:05.713 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:43:06.074 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:43:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:43:09.789 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:43:20.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-10T20:43:31.612 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:43:31.636 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:43:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:43:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:43:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:43:32.926 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:43:32.926 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-10T20:43:32.926 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-10T20:43:32.926 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:43:32.926 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:43:32.927 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-10T20:43:32.927 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-10T20:43:33.079 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:43:33.297 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:43:33.297 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:43:33.297 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:43:33.297 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:43:33.298 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:43:33.298 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T20:43:33.298 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:43:33.298 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:43:33.512 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:43:33.512 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:43:33.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2024-04-10T20:43:33.545 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2024-04-10T20:43:33.545 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2024-04-10T20:43:33.545 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.02 2024-04-10T20:43:33.545 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 129.el8 2024-04-10T20:43:33.545 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2024-04-10T20:43:33.545 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:43:33.545 INFO:teuthology.orchestra.run.smithi187.stdout:Group : System Environment/Base 2024-04-10T20:43:33.545 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 9265586 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:Relocations : (not relocatable) 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : CentOS BuildSystem 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2024-04-10T20:43:33.546 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-10T20:43:33.547 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:43:33.547 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:43:33.547 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:43:33.547 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2024-04-10T20:43:33.547 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-10T20:43:33.547 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:43:33.548 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:43:33.548 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:43:33.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:43:34.791 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2024-04-10T20:43:36.703 INFO:teuthology.orchestra.run.smithi187.stderr:done 2024-04-10T20:43:36.705 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-04-10T20:43:36.768 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.6zm05ymWza 2024-04-10T20:43:36.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6zm05ymWza 2024-04-10T20:43:36.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.6zm05ymWza 2024-04-10T20:43:37.000 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.6zm05ymWza is 6KB 2024-04-10T20:43:37.047 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.6zm05ymWza 2024-04-10T20:43:37.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:43:37.133 INFO:teuthology.orchestra.run.smithi187.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:43:37.134 INFO:teuthology.orchestra.run.smithi187.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:43:37.134 INFO:teuthology.orchestra.run.smithi187.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:43:37.134 INFO:teuthology.orchestra.run.smithi187.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:43:37.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:43:37.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2024-04-10T20:43:38.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T20:43:38.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:43:39.284 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-04-10T20:43:39.586 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T20:43:39.587 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:43:39.587 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-04-10T20:43:39.604 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:43:39.605 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:43:39.605 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:43:39.605 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:43:40.605 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T20:43:40.606 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:43:40.675 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-04-10T20:43:40.690 DEBUG:teuthology.parallel:result is None 2024-04-10T20:44:07.292 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T20:44:07.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-10T20:44:07.293 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:44:16.414 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 68 kB/s | 16 MB 04:06 2024-04-10T20:44:20.173 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 364 kB/s | 24 kB 00:00 2024-04-10T20:44:20.319 INFO:teuthology.orchestra.run.smithi161.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T20:44:23.186 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T20:44:23.186 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T20:44:23.316 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-10T20:44:23.317 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-10T20:44:23.318 INFO:teuthology.orchestra.run.smithi161.stdout:Install 3 Packages 2024-04-10T20:44:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-10T20:44:23.318 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 90 M 2024-04-10T20:44:23.318 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 96 M 2024-04-10T20:44:23.318 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-10T20:44:25.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-04-10T20:44:28.740 INFO:teuthology.orchestra.run.smithi161.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 2.0 MB/s | 10 MB 00:05 2024-04-10T20:44:34.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-10T20:44:34.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:44:35.598 INFO:teuthology.orchestra.run.smithi161.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 3.0 MB/s | 36 MB 00:12 2024-04-10T20:44:37.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-04-10T20:44:43.422 INFO:teuthology.orchestra.run.smithi161.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 2.2 MB/s | 44 MB 00:19 2024-04-10T20:44:43.422 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-10T20:44:43.422 INFO:teuthology.orchestra.run.smithi161.stdout:Total 4.5 MB/s | 90 MB 00:20 2024-04-10T20:44:44.103 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-10T20:44:44.559 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-10T20:44:44.559 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-10T20:44:45.822 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-10T20:44:45.853 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-10T20:44:48.656 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-10T20:44:49.286 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:44:49.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-10T20:44:49.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:44:52.882 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:44:53.277 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:44:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:44:57.035 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:45:08.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-04-10T20:45:19.169 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:45:19.194 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:45:19.663 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:45:19.663 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:45:19.663 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:45:20.120 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:45:20.120 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-10T20:45:20.120 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-10T20:45:20.121 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:45:20.121 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:45:20.121 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-10T20:45:20.121 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-10T20:45:20.287 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:45:20.506 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:45:20.507 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:45:20.507 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:45:20.507 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:45:20.507 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:45:20.508 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T20:45:20.508 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi161.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:45:20.508 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:45:20.721 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:45:20.721 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:45:20.722 DEBUG:teuthology.orchestra.run.smithi161:> sudo rpm -qi grub2-tools 2024-04-10T20:45:20.756 INFO:teuthology.orchestra.run.smithi161.stdout:Name : grub2-tools 2024-04-10T20:45:20.757 INFO:teuthology.orchestra.run.smithi161.stdout:Epoch : 1 2024-04-10T20:45:20.757 INFO:teuthology.orchestra.run.smithi161.stdout:Version : 2.02 2024-04-10T20:45:20.757 INFO:teuthology.orchestra.run.smithi161.stdout:Release : 129.el8 2024-04-10T20:45:20.757 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: x86_64 2024-04-10T20:45:20.757 INFO:teuthology.orchestra.run.smithi161.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:45:20.757 INFO:teuthology.orchestra.run.smithi161.stdout:Group : System Environment/Base 2024-04-10T20:45:20.757 INFO:teuthology.orchestra.run.smithi161.stdout:Size : 9265586 2024-04-10T20:45:20.758 INFO:teuthology.orchestra.run.smithi161.stdout:License : GPLv3+ 2024-04-10T20:45:20.758 INFO:teuthology.orchestra.run.smithi161.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:45:20.758 INFO:teuthology.orchestra.run.smithi161.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:45:20.758 INFO:teuthology.orchestra.run.smithi161.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:45:20.758 INFO:teuthology.orchestra.run.smithi161.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:45:20.758 INFO:teuthology.orchestra.run.smithi161.stdout:Relocations : (not relocatable) 2024-04-10T20:45:20.758 INFO:teuthology.orchestra.run.smithi161.stdout:Packager : CentOS BuildSystem 2024-04-10T20:45:20.759 INFO:teuthology.orchestra.run.smithi161.stdout:Vendor : CentOS 2024-04-10T20:45:20.759 INFO:teuthology.orchestra.run.smithi161.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:45:20.759 INFO:teuthology.orchestra.run.smithi161.stdout:Summary : Support tools for GRUB. 2024-04-10T20:45:20.759 INFO:teuthology.orchestra.run.smithi161.stdout:Description : 2024-04-10T20:45:20.759 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-10T20:45:20.759 INFO:teuthology.orchestra.run.smithi161.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:45:20.759 INFO:teuthology.orchestra.run.smithi161.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:45:20.759 INFO:teuthology.orchestra.run.smithi161.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:45:20.759 INFO:teuthology.orchestra.run.smithi161.stdout:hardware devices. 2024-04-10T20:45:20.760 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-10T20:45:20.760 INFO:teuthology.orchestra.run.smithi161.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:45:20.761 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:45:20.761 INFO:teuthology.task.kernel:Updating grub on smithi161 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:45:20.761 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:45:22.267 INFO:teuthology.orchestra.run.smithi161.stderr:Generating grub configuration file ... 2024-04-10T20:45:23.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-10T20:45:23.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:45:24.363 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-04-10T20:45:24.418 INFO:teuthology.orchestra.run.smithi161.stderr:done 2024-04-10T20:45:24.419 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-04-10T20:45:24.450 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.fTFjMSdpVN 2024-04-10T20:45:24.451 DEBUG:teuthology.orchestra.run.smithi161:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fTFjMSdpVN 2024-04-10T20:45:24.524 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.fTFjMSdpVN 2024-04-10T20:45:24.675 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.fTFjMSdpVN is 6KB 2024-04-10T20:45:24.722 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.fTFjMSdpVN 2024-04-10T20:45:24.740 DEBUG:teuthology.orchestra.run.smithi161:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:45:24.812 INFO:teuthology.orchestra.run.smithi161.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:45:24.812 INFO:teuthology.orchestra.run.smithi161.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:45:24.812 INFO:teuthology.orchestra.run.smithi161.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:45:24.812 INFO:teuthology.orchestra.run.smithi161.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:45:24.814 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:45:24.974 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-10T20:45:24.974 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:45:24.975 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-04-10T20:45:24.990 DEBUG:teuthology.orchestra.run.smithi161:> sudo shutdown -r now 2024-04-10T20:45:25.035 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:45:25.036 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:45:25.036 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:45:25.036 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:45:26.036 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T20:45:26.037 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:45:26.113 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2024-04-10T20:45:26.130 DEBUG:teuthology.parallel:result is None 2024-04-10T20:45:54.993 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T20:45:54.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-10T20:45:54.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:46:13.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-04-10T20:46:22.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-10T20:46:22.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:46:25.633 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-04-10T20:46:37.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-10T20:46:37.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:46:56.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-04-10T20:47:11.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-10T20:47:11.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:47:14.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-04-10T20:47:32.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-10T20:47:32.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:47:32.407 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-04-10T20:47:32.691 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-10T20:47:32.691 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:47:32.692 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-04-10T20:47:32.711 INFO:teuthology.orchestra.run.smithi161.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:47:32.711 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:47:32.711 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:47:32.711 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:47:33.712 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T20:47:33.712 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:47:33.780 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2024-04-10T20:47:33.794 DEBUG:teuthology.parallel:result is None 2024-04-10T20:47:33.794 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T20:47:33.803 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T20:47:33.803 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:47:33.807 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:47:33.809 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:47:33.832 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T20:47:33.848 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T20:47:33.897 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T20:47:33.898 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:47:33.900 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:47:33.902 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:47:33.947 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T20:47:33.953 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T20:47:33.954 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:47:33.983 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:47:33.986 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:47:34.018 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:47:34.019 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:47:34.024 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:47:34.029 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:47:34.030 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:47:34.036 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:47:34.037 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T20:47:34.044 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T20:47:34.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:47:34.073 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:47:34.076 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:47:34.108 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T20:47:34.117 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T20:47:34.117 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:47:34.143 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:47:34.146 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:47:34.167 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:47:34.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:47:34.306 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:47:34.360 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:47:34.392 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-10T20:47:34.392 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:47:34.458 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:47:34.517 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:47:34.564 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:47:34.661 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:47:34.695 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-10T20:47:34.695 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:47:34.764 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:47:34.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:47:34.871 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:47:34.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:47:34.961 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-10T20:47:34.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:47:35.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-04-10T20:47:35.032 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-04-10T20:47:35.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-04-10T20:47:35.095 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:47:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:47:35.136 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:47:35.564 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T20:47:35.572 INFO:teuthology.task.internal:Starting timer... 2024-04-10T20:47:35.572 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T20:47:35.618 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T20:47:35.666 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T20:47:35.666 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-04-10T20:47:35.724 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-04-10T20:47:35.725 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-04-10T20:47:35.831 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-04-10T20:47:35.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2024-04-10T20:47:35.931 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2024-04-10T20:47:35.932 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T20:47:35.932 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-04-10T20:47:36.007 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-04-10T20:47:36.007 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-04-10T20:47:36.036 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-04-10T20:47:36.036 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2024-04-10T20:47:36.179 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2024-04-10T20:47:36.180 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi161.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2024-04-10T20:47:36.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:47:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:47:36.207 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:47:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:47:36.236 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:47:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:47:36.263 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T20:47:36.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:47:36.289 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:47:36.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:47:36.344 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T20:47:36.354 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T20:47:36.360 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T20:47:36.537 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T20:47:36.552 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T20:47:36.553 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi161.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T21:05:57.565 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203/ansible_failures.yaml 2024-04-10T21:05:57.567 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T21:05:57.780 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9fde262ded7a46acbc86f950ea5aa223 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T21:05:57.781 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T21:05:57.789 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T21:05:57.790 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T21:05:57.856 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T21:05:57.923 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T21:05:57.990 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T21:05:58.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T21:05:58.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T21:05:58.085 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T21:05:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T21:05:58.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T21:05:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T21:05:58.149 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T21:05:58.156 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T21:05:58.197 INFO:teuthology.task.internal:Duration was 1102.624789 seconds 2024-04-10T21:05:58.197 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T21:05:58.247 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T21:05:58.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T21:05:58.250 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T21:05:58.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T21:05:58.299 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T21:05:58.301 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T21:05:58.301 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T21:05:58.667 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T21:05:58.667 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-04-10T21:05:58.668 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T21:05:58.694 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-04-10T21:05:58.694 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T21:05:58.718 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-04-10T21:05:58.719 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T21:05:58.742 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T21:05:58.742 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T21:05:58.745 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T21:05:58.761 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T21:05:58.822 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T21:05:58.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T21:05:58.829 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T21:05:58.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T21:05:58.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T21:05:58.933 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T21:05:58.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T21:05:58.961 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T21:05:58.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T21:05:59.014 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T21:05:59.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T21:05:59.025 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T21:05:59.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T21:05:59.049 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-04-10T21:05:59.053 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-04-10T21:05:59.080 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-04-10T21:05:59.108 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T21:05:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T21:05:59.134 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T21:05:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T21:05:59.151 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T21:05:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T21:05:59.168 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T21:05:59.176 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T21:05:59.178 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203/remote/smithi088 2024-04-10T21:05:59.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T21:05:59.234 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203/remote/smithi161 2024-04-10T21:05:59.235 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T21:05:59.282 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203/remote/smithi187 2024-04-10T21:05:59.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T21:05:59.339 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T21:05:59.339 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T21:05:59.341 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T21:05:59.344 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T21:05:59.361 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T21:05:59.372 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T21:05:59.372 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T21:05:59.435 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T21:05:59.435 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T21:05:59.438 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T21:05:59.440 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T21:05:59.452 INFO:teuthology.orchestra.run.smithi088.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 21:05 /home/ubuntu/cephtest 2024-04-10T21:05:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: 525355 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 21:05 /home/ubuntu/cephtest 2024-04-10T21:05:59.457 INFO:teuthology.orchestra.run.smithi187.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 21:05 /home/ubuntu/cephtest 2024-04-10T21:05:59.458 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T21:05:59.548 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T21:05:59.667 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T21:05:59.694 DEBUG:teuthology.nuke:shortname: smithi088 2024-04-10T21:05:59.694 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T21:05:59.716 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:26:36.006686', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJt6LKF7+imv80NqA/P4Msf0C5EwDQIqLwTQTEIlwH+kqnllR+q9GBuvrnPkwdfUEEMl1zRK9WvEwWhGBhjO0c='} 2024-04-10T21:05:59.720 DEBUG:teuthology.nuke:shortname: smithi161 2024-04-10T21:05:59.720 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T21:05:59.743 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:26:36.004711', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwnOZA3Tg0HffDpwLQTQqn3CyIMvS/EOob7YyfWGnybPWQM9sS6rozkWndBJS9IgYaOOSO+1ks6tMM2b4HBc+o='} 2024-04-10T21:05:59.746 DEBUG:teuthology.nuke:shortname: smithi187 2024-04-10T21:05:59.747 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T21:05:59.763 INFO:teuthology.orchestra.console.smithi088:Power off 2024-04-10T21:05:59.763 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T21:05:59.773 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:26:36.008226', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+tbx6m0sf6E0EWToc5AFdYRYf6GMe6grZUxN4XzMSKVyaZhjKz3yDh+q/RC/JzzjOXzBc+GLXGRL0ZebyyQ0I='} 2024-04-10T21:05:59.785 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2024-04-10T21:05:59.785 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:05:59.800 INFO:teuthology.orchestra.console.smithi161:Power off 2024-04-10T21:05:59.800 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T21:05:59.813 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-04-10T21:05:59.821 DEBUG:teuthology.orchestra.console.smithi161:power off output: Chassis Power Control: Down/Off 2024-04-10T21:05:59.822 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:05:59.839 INFO:teuthology.orchestra.console.smithi187:Power off 2024-04-10T21:05:59.839 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T21:05:59.846 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-04-10T21:05:59.861 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2024-04-10T21:05:59.861 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:05:59.884 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-04-10T21:06:03.813 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:06:03.847 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:06:03.884 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:06:03.923 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-04-10T21:06:03.956 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-04-10T21:06:03.994 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-04-10T21:06:07.928 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:06:07.956 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:06:07.995 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:06:08.038 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-04-10T21:06:08.065 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-04-10T21:06:08.106 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-04-10T21:06:12.042 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:06:12.066 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:06:12.107 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T21:06:12.153 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2024-04-10T21:06:12.176 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is off 2024-04-10T21:06:12.216 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2024-04-10T21:06:12.254 INFO:teuthology.orchestra.console.smithi088:Power off completed 2024-04-10T21:06:12.277 INFO:teuthology.orchestra.console.smithi161:Power off completed 2024-04-10T21:06:12.317 INFO:teuthology.orchestra.console.smithi187:Power off completed 2024-04-10T21:06:12.423 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} duration: 1102.6247889995575 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9fde262ded7a46acbc86f950ea5aa223 status: dead success: false 2024-04-10T21:06:12.423 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T21:06:12.510 INFO:teuthology.run:DEAD