2024-04-10T18:42:01.116 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T18:42:01.116 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964 --verbose --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} --name yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964/orig.config.yaml 2024-04-10T18:42:01.163 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T18:42:01.263 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964 branch: wip-yuri10-testing-2024-03-24-1159 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} email: yweinste@redhat.com first_in_suite: false job_id: '7650964' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-03-24-1159 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - must scrub before tier agent can activate sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-yuri10-testing-2024-03-24-1159 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 8148 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-03-24-1159 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d targets: smithi042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEl1i8WX1Ko2umMFieFEYt+gSb/TkAHDhIUCnmpspZGL8k1S1veDgICVGyCDtB+l+z7V61XAYTjmQPgs1aLcud0= smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI/n8ijkP3hAhBmUmRxMZ/F4RVDw5jmKZT0xvhN+QkxrG2aFADvtClcqyoXPDgBp8e0QFQ131hSIVaQ9OT1/44c= smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF5Nc+nE5t/6ejaWrpuQGqQSNHGMiW0qUe8ueumrm5RCyOAEU/1HG2IRKtLYy8UbbetzIvtDu4qfpv/AQsAWHaw= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: reef exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 3600 - sudo ceph osd pool set cache target_max_objects 250 - sudo ceph osd pool set cache min_read_recency_for_promote 2 - rados: clients: - client.2 objects: 500 op_weights: cache_evict: 50 cache_flush: 50 cache_try_flush: 50 copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:20:47 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T18:42:01.264 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa; will attempt to use it 2024-04-10T18:42:01.264 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks 2024-04-10T18:42:01.265 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T18:42:01.266 INFO:teuthology.task.internal:Saving configuration 2024-04-10T18:42:01.282 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T18:42:01.289 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T18:42:01.314 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 18:37:09.225074', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEl1i8WX1Ko2umMFieFEYt+gSb/TkAHDhIUCnmpspZGL8k1S1veDgICVGyCDtB+l+z7V61XAYTjmQPgs1aLcud0='} 2024-04-10T18:42:01.339 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 18:37:09.222029', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI/n8ijkP3hAhBmUmRxMZ/F4RVDw5jmKZT0xvhN+QkxrG2aFADvtClcqyoXPDgBp8e0QFQ131hSIVaQ9OT1/44c='} 2024-04-10T18:42:01.359 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 18:37:09.223617', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF5Nc+nE5t/6ejaWrpuQGqQSNHGMiW0qUe8ueumrm5RCyOAEU/1HG2IRKtLYy8UbbetzIvtDu4qfpv/AQsAWHaw='} 2024-04-10T18:42:01.359 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T18:42:01.366 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T18:42:01.366 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T18:42:01.366 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T18:42:01.366 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T18:42:01.515 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1505804160>, signals=[15]) 2024-04-10T18:42:01.515 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T18:42:01.521 INFO:teuthology.task.internal:Opening connections... 2024-04-10T18:42:01.522 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2024-04-10T18:42:01.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T18:42:01.600 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-04-10T18:42:01.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T18:42:01.670 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-04-10T18:42:01.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T18:42:01.740 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T18:42:01.747 DEBUG:teuthology.orchestra.run.smithi042:> uname -m 2024-04-10T18:42:01.770 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2024-04-10T18:42:01.770 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2024-04-10T18:42:01.831 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="CentOS Stream" 2024-04-10T18:42:01.831 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="8" 2024-04-10T18:42:01.831 INFO:teuthology.orchestra.run.smithi042.stdout:ID="centos" 2024-04-10T18:42:01.831 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE="rhel fedora" 2024-04-10T18:42:01.831 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="8" 2024-04-10T18:42:01.831 INFO:teuthology.orchestra.run.smithi042.stdout:PLATFORM_ID="platform:el8" 2024-04-10T18:42:01.831 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T18:42:01.832 INFO:teuthology.orchestra.run.smithi042.stdout:ANSI_COLOR="0;31" 2024-04-10T18:42:01.832 INFO:teuthology.orchestra.run.smithi042.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T18:42:01.832 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://centos.org/" 2024-04-10T18:42:01.832 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T18:42:01.832 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T18:42:01.832 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T18:42:01.832 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2024-04-10T18:42:01.853 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-04-10T18:42:01.876 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-04-10T18:42:01.876 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="8" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="8" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el8" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T18:42:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T18:42:01.939 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-04-10T18:42:01.961 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-04-10T18:42:01.983 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-04-10T18:42:01.983 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-04-10T18:42:02.045 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-04-10T18:42:02.045 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8" 2024-04-10T18:42:02.045 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-04-10T18:42:02.045 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-04-10T18:42:02.045 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8" 2024-04-10T18:42:02.046 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8" 2024-04-10T18:42:02.046 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T18:42:02.046 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-04-10T18:42:02.046 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T18:42:02.046 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-04-10T18:42:02.046 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T18:42:02.046 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T18:42:02.046 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T18:42:02.046 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-04-10T18:42:02.078 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T18:42:02.087 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T18:42:02.132 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T18:42:02.132 DEBUG:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2024-04-10T18:42:02.134 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-04-10T18:42:02.136 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-04-10T18:42:02.157 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T18:42:02.163 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T18:42:02.163 DEBUG:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2024-04-10T18:42:02.196 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-04-10T18:42:02.198 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-04-10T18:42:02.226 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T18:42:02.293 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T18:42:02.306 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T18:42:02.306 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T18:42:02.306 DEBUG:teuthology.orchestra.run.smithi042:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T18:42:02.307 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T18:42:02.307 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T18:42:02.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T18:42:02.329 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-04-10T18:42:02.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T18:42:02.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T18:42:02.330 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-10T18:42:02.331 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-04-10T18:42:02.391 INFO:teuthology.orchestra.run.smithi042.stdout:4.18.0-489.el8.x86_64 2024-04-10T18:42:02.391 INFO:teuthology.task.kernel:Running kernel on smithi042: 4.18.0-489.el8.x86_64 2024-04-10T18:42:02.391 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum install -y kernel 2024-04-10T18:42:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:4.18.0-489.el8.x86_64 2024-04-10T18:42:02.393 INFO:teuthology.orchestra.run.smithi062.stdout:4.18.0-489.el8.x86_64 2024-04-10T18:42:02.393 INFO:teuthology.task.kernel:Running kernel on smithi045: 4.18.0-489.el8.x86_64 2024-04-10T18:42:02.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2024-04-10T18:42:02.394 INFO:teuthology.task.kernel:Running kernel on smithi062: 4.18.0-489.el8.x86_64 2024-04-10T18:42:02.394 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum install -y kernel 2024-04-10T18:42:13.026 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 8 - AppStream 3.5 MB/s | 28 MB 00:07 2024-04-10T18:42:21.710 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 8 - BaseOS 3.0 MB/s | 10 MB 00:03 2024-04-10T18:44:16.494 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - AppStream 217 kB/s | 28 MB 02:11 2024-04-10T18:44:19.356 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - AppStream 212 kB/s | 28 MB 02:14 2024-04-10T18:44:33.197 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 8 - Extras 140 B/s | 18 kB 02:09 2024-04-10T18:45:17.994 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 8 - Extras common packages 176 B/s | 7.7 kB 00:44 2024-04-10T18:46:35.866 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - BaseOS 79 kB/s | 10 MB 02:14 2024-04-10T18:46:36.916 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - BaseOS 80 kB/s | 10 MB 02:12 2024-04-10T18:46:39.616 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - Extras 19 kB/s | 18 kB 00:00 2024-04-10T18:46:46.460 INFO:teuthology.orchestra.run.smithi042.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T18:46:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T18:46:46.461 INFO:teuthology.orchestra.run.smithi042.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T18:46:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T18:47:21.630 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - Extras common packages 188 B/s | 7.7 kB 00:41 2024-04-10T18:48:46.080 INFO:teuthology.orchestra.run.smithi045.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:24 2024-04-10T18:48:46.080 INFO:teuthology.orchestra.run.smithi045.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T18:48:46.081 INFO:teuthology.orchestra.run.smithi045.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T18:48:46.081 INFO:teuthology.orchestra.run.smithi045.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T18:48:49.429 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - Extras 138 B/s | 18 kB 02:11 2024-04-10T18:50:45.234 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 140 kB/s | 16 MB 01:59 2024-04-10T18:50:49.342 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 135 kB/s | 24 kB 00:00 2024-04-10T18:50:49.493 INFO:teuthology.orchestra.run.smithi045.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T18:50:52.514 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T18:50:52.514 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T18:50:52.634 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-10T18:50:52.635 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-10T18:50:52.635 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-04-10T18:50:52.635 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-10T18:50:52.635 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-10T18:50:52.635 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T18:50:52.635 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-10T18:50:52.635 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T18:50:52.636 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T18:50:52.636 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-10T18:50:52.636 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-10T18:50:52.636 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-10T18:50:52.636 INFO:teuthology.orchestra.run.smithi045.stdout:Install 3 Packages 2024-04-10T18:50:52.636 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-10T18:50:52.636 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 90 M 2024-04-10T18:50:52.636 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 96 M 2024-04-10T18:50:52.637 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-10T18:51:40.446 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - Extras common packages 46 B/s | 7.7 kB 02:50 2024-04-10T18:53:07.569 INFO:teuthology.orchestra.run.smithi045.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 3.0 MB/s | 10 MB 00:03 2024-04-10T18:53:08.710 INFO:teuthology.orchestra.run.smithi062.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T18:53:08.711 INFO:teuthology.orchestra.run.smithi062.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T18:53:08.711 INFO:teuthology.orchestra.run.smithi062.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T18:53:08.711 INFO:teuthology.orchestra.run.smithi062.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T18:53:09.794 INFO:teuthology.orchestra.run.smithi045.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 7.7 MB/s | 44 MB 00:05 2024-04-10T18:53:10.278 INFO:teuthology.orchestra.run.smithi045.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 5.9 MB/s | 36 MB 00:06 2024-04-10T18:53:10.278 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-10T18:53:10.278 INFO:teuthology.orchestra.run.smithi045.stdout:Total 673 kB/s | 90 MB 02:17 2024-04-10T18:53:10.996 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-10T18:53:11.436 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-10T18:53:11.436 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-10T18:53:12.752 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-10T18:53:12.779 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-10T18:53:15.782 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-10T18:53:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T18:53:20.139 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T18:53:20.533 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T18:53:24.216 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T18:53:24.505 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:53:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:53:48.258 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:53:48.618 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:53:48.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T18:53:48.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T18:53:49.113 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:53:49.113 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-10T18:53:49.113 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-10T18:53:49.113 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T18:53:49.113 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T18:53:49.113 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-10T18:53:49.113 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-10T18:53:49.274 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T18:53:49.499 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T18:53:49.500 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T18:53:49.500 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T18:53:49.500 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T18:53:49.500 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T18:53:49.500 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T18:53:49.500 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T18:53:49.501 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T18:53:49.722 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T18:53:49.722 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T18:53:49.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-04-10T18:53:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-04-10T18:53:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-04-10T18:53:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.02 2024-04-10T18:53:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 129.el8 2024-04-10T18:53:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-04-10T18:53:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T18:53:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:Group : System Environment/Base 2024-04-10T18:53:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 9265586 2024-04-10T18:53:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Relocations : (not relocatable) 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : CentOS BuildSystem 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T18:53:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-04-10T18:53:49.758 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-10T18:53:49.758 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T18:53:49.758 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T18:53:49.759 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 4.18.0-552.el8.x86_64 2024-04-10T18:53:49.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T18:53:51.030 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-04-10T18:53:53.156 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-04-10T18:53:53.158 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-04-10T18:53:53.200 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.nblnYtCzcq 2024-04-10T18:53:53.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nblnYtCzcq 2024-04-10T18:53:53.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.nblnYtCzcq 2024-04-10T18:53:53.416 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.nblnYtCzcq is 6KB 2024-04-10T18:53:53.464 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.nblnYtCzcq 2024-04-10T18:53:53.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T18:53:53.560 INFO:teuthology.orchestra.run.smithi045.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T18:53:53.560 INFO:teuthology.orchestra.run.smithi045.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T18:53:53.560 INFO:teuthology.orchestra.run.smithi045.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T18:53:53.560 INFO:teuthology.orchestra.run.smithi045.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T18:53:53.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T18:53:53.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-04-10T18:53:56.437 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 349 kB/s | 16 MB 00:47 2024-04-10T18:54:00.462 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 347 kB/s | 24 kB 00:00 2024-04-10T18:54:00.613 INFO:teuthology.orchestra.run.smithi062.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T18:54:03.589 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T18:54:03.590 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T18:54:03.695 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-10T18:54:03.696 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-10T18:54:03.697 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-10T18:54:03.697 INFO:teuthology.orchestra.run.smithi062.stdout:Install 3 Packages 2024-04-10T18:54:03.697 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-10T18:54:03.697 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 90 M 2024-04-10T18:54:03.697 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 96 M 2024-04-10T18:54:03.697 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-10T18:54:23.744 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T18:54:23.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-10T18:54:23.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T18:54:42.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-04-10T18:54:51.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-10T18:54:51.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T18:55:51.210 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T18:56:03.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-10T18:56:03.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T18:56:03.572 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-10T18:56:03.839 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-10T18:56:03.839 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T18:56:03.839 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-10T18:56:03.858 INFO:teuthology.orchestra.run.smithi045.stdout:4.18.0-552.el8.x86_64 2024-04-10T18:56:03.859 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T18:56:03.859 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T18:56:03.859 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T18:56:04.859 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T18:56:04.859 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T18:56:04.957 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-04-10T18:56:04.970 DEBUG:teuthology.parallel:result is None 2024-04-10T18:56:16.339 INFO:teuthology.orchestra.run.smithi062.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 6.6 MB/s | 10 MB 00:01 2024-04-10T18:56:18.381 INFO:teuthology.orchestra.run.smithi062.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 10 MB/s | 36 MB 00:03 2024-04-10T18:56:19.090 INFO:teuthology.orchestra.run.smithi062.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 10 MB/s | 44 MB 00:04 2024-04-10T18:56:19.090 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-10T18:56:19.090 INFO:teuthology.orchestra.run.smithi062.stdout:Total 684 kB/s | 90 MB 02:15 2024-04-10T18:56:19.808 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-10T18:56:20.264 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-10T18:56:20.264 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-10T18:56:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-10T18:56:21.609 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-10T18:56:24.574 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-10T18:56:25.127 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T18:56:28.953 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T18:56:29.343 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T18:56:33.032 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T18:56:33.306 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:56:56.670 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:56:56.694 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:56:57.319 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:56:57.320 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T18:56:57.320 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T18:56:57.815 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T18:56:57.815 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-10T18:56:57.815 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-10T18:56:57.815 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T18:56:57.815 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T18:56:57.815 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-10T18:56:57.815 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-10T18:56:57.960 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T18:56:58.185 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T18:56:58.185 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T18:56:58.185 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T18:56:58.185 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T18:56:58.185 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T18:56:58.185 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T18:56:58.186 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T18:56:58.186 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T18:56:58.408 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T18:56:58.408 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T18:56:58.408 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.02 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 129.el8 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Group : System Environment/Base 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 9265586 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T18:56:58.443 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:Relocations : (not relocatable) 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : CentOS BuildSystem 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T18:56:58.444 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2024-04-10T18:56:58.445 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-10T18:56:58.445 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T18:56:58.445 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T18:56:58.446 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 4.18.0-552.el8.x86_64 2024-04-10T18:56:58.446 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T18:56:59.728 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-04-10T18:57:01.850 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-04-10T18:57:01.852 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-04-10T18:57:01.876 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.DaOLvery9Z 2024-04-10T18:57:01.876 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DaOLvery9Z 2024-04-10T18:57:01.954 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.DaOLvery9Z 2024-04-10T18:57:02.115 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.DaOLvery9Z is 6KB 2024-04-10T18:57:02.163 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.DaOLvery9Z 2024-04-10T18:57:02.180 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T18:57:02.262 INFO:teuthology.orchestra.run.smithi062.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T18:57:02.263 INFO:teuthology.orchestra.run.smithi062.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T18:57:02.263 INFO:teuthology.orchestra.run.smithi062.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T18:57:02.263 INFO:teuthology.orchestra.run.smithi062.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T18:57:02.264 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T18:57:02.441 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-04-10T18:57:32.443 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T18:57:32.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-10T18:57:32.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T18:58:32.446 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T18:58:41.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-10T18:58:41.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T18:58:41.822 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-04-10T18:58:42.085 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-10T18:58:42.085 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T18:58:42.085 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-04-10T18:58:42.103 INFO:teuthology.orchestra.run.smithi062.stdout:4.18.0-552.el8.x86_64 2024-04-10T18:58:42.103 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T18:58:42.103 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T18:58:42.103 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T18:58:43.103 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T18:58:43.104 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T18:58:43.194 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2024-04-10T18:58:43.209 DEBUG:teuthology.parallel:result is None 2024-04-10T18:59:54.748 INFO:teuthology.orchestra.run.smithi042.stdout:Extra Packages for Enterprise Linux 21 kB/s | 16 MB 13:08 2024-04-10T18:59:58.690 INFO:teuthology.orchestra.run.smithi042.stdout:lab-extras 406 kB/s | 24 kB 00:00 2024-04-10T18:59:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:00:01.804 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:00:01.805 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:00:01.923 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-04-10T19:00:01.923 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-04-10T19:00:01.923 INFO:teuthology.orchestra.run.smithi042.stdout: Package Architecture Version Repository Size 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout:Install 3 Packages 2024-04-10T19:00:01.924 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-10T19:00:01.925 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 90 M 2024-04-10T19:00:01.925 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 96 M 2024-04-10T19:00:01.925 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-04-10T19:02:16.821 INFO:teuthology.orchestra.run.smithi042.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 3.0 MB/s | 10 MB 00:03 2024-04-10T19:02:17.196 INFO:teuthology.orchestra.run.smithi042.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 9.6 MB/s | 36 MB 00:03 2024-04-10T19:02:18.388 INFO:teuthology.orchestra.run.smithi042.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 8.8 MB/s | 44 MB 00:04 2024-04-10T19:02:18.388 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:02:18.389 INFO:teuthology.orchestra.run.smithi042.stdout:Total 679 kB/s | 90 MB 02:16 2024-04-10T19:02:19.104 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-04-10T19:02:19.554 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-04-10T19:02:19.555 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-04-10T19:02:20.864 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-04-10T19:02:20.894 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-04-10T19:02:23.858 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-04-10T19:02:24.407 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:02:28.192 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:02:28.599 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:02:32.279 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:02:32.568 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:02:56.057 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:02:56.080 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:02:56.448 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:02:56.448 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:02:56.448 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:02:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:02:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-10T19:02:56.939 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-04-10T19:02:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T19:02:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T19:02:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-10T19:02:56.940 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-04-10T19:02:57.099 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:02:57.323 INFO:teuthology.orchestra.run.smithi042.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:02:57.323 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:02:57.323 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T19:02:57.324 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T19:02:57.324 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T19:02:57.324 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T19:02:57.324 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi042.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T19:02:57.324 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:02:57.542 INFO:teuthology.orchestra.run.smithi042.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:02:57.543 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:02:57.543 DEBUG:teuthology.orchestra.run.smithi042:> sudo rpm -qi grub2-tools 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:Name : grub2-tools 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:Epoch : 1 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:Version : 2.02 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:Release : 129.el8 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: x86_64 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:Group : System Environment/Base 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:Size : 9265586 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:License : GPLv3+ 2024-04-10T19:02:57.578 INFO:teuthology.orchestra.run.smithi042.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:Relocations : (not relocatable) 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:Packager : CentOS BuildSystem 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:Vendor : CentOS 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:Summary : Support tools for GRUB. 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:Description : 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout:hardware devices. 2024-04-10T19:02:57.579 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-10T19:02:57.580 INFO:teuthology.orchestra.run.smithi042.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T19:02:57.580 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T19:02:57.580 INFO:teuthology.task.kernel:Updating grub on smithi042 to boot 4.18.0-552.el8.x86_64 2024-04-10T19:02:57.580 DEBUG:teuthology.orchestra.run.smithi042:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T19:02:59.039 INFO:teuthology.orchestra.run.smithi042.stderr:Generating grub configuration file ... 2024-04-10T19:03:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:done 2024-04-10T19:03:01.174 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-04-10T19:03:01.197 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.vKZ90eMJHK 2024-04-10T19:03:01.197 DEBUG:teuthology.orchestra.run.smithi042:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vKZ90eMJHK 2024-04-10T19:03:01.290 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.vKZ90eMJHK 2024-04-10T19:03:01.393 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.vKZ90eMJHK is 6KB 2024-04-10T19:03:01.441 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.vKZ90eMJHK 2024-04-10T19:03:01.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T19:03:01.533 INFO:teuthology.orchestra.run.smithi042.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T19:03:01.533 INFO:teuthology.orchestra.run.smithi042.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T19:03:01.533 INFO:teuthology.orchestra.run.smithi042.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T19:03:01.534 INFO:teuthology.orchestra.run.smithi042.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T19:03:01.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T19:03:01.698 DEBUG:teuthology.orchestra.run.smithi042:> sudo shutdown -r now 2024-04-10T19:03:31.701 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T19:03:31.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-04-10T19:03:31.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:03:50.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.42 2024-04-10T19:03:59.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-04-10T19:03:59.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:04:59.050 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T19:05:11.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-04-10T19:05:11.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:05:11.393 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-04-10T19:05:11.695 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-04-10T19:05:11.695 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T19:05:11.695 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-04-10T19:05:11.757 INFO:teuthology.orchestra.run.smithi042.stdout:4.18.0-552.el8.x86_64 2024-04-10T19:05:11.757 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T19:05:11.757 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T19:05:11.757 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T19:05:12.758 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T19:05:12.758 DEBUG:teuthology.orchestra.run.smithi042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T19:05:12.837 INFO:teuthology.orchestra.run.smithi042.stdout:ttyS1 2024-04-10T19:05:12.851 DEBUG:teuthology.parallel:result is None 2024-04-10T19:05:12.851 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T19:05:12.858 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T19:05:12.858 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T19:05:12.861 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T19:05:12.864 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T19:05:12.889 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T19:05:12.921 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T19:05:12.954 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T19:05:12.954 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T19:05:12.957 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T19:05:12.959 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T19:05:12.996 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T19:05:13.002 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T19:05:13.002 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T19:05:13.022 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T19:05:13.025 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T19:05:13.057 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:05:13.078 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:05:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:05:13.090 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:05:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:05:13.101 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T19:05:13.108 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T19:05:13.108 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T19:05:13.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T19:05:13.115 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T19:05:13.149 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T19:05:13.158 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T19:05:13.158 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T19:05:13.191 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T19:05:13.194 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T19:05:13.213 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:05:13.390 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:05:13.436 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:05:13.553 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:05:13.583 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-10T19:05:13.583 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T19:05:13.652 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:05:13.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:05:13.816 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:05:13.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:05:14.009 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-10T19:05:14.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T19:05:14.082 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:05:14.199 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:05:14.239 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:05:14.399 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:05:14.433 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-10T19:05:14.433 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T19:05:14.505 DEBUG:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2024-04-10T19:05:14.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-04-10T19:05:14.511 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-04-10T19:05:14.572 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:05:14.576 INFO:teuthology.orchestra.run.smithi042.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:05:14.626 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:05:14.999 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T19:05:15.007 INFO:teuthology.task.internal:Starting timer... 2024-04-10T19:05:15.007 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T19:05:15.049 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T19:05:15.081 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T19:05:15.082 DEBUG:teuthology.orchestra.run.smithi042:> sudo service auditd rotate 2024-04-10T19:05:15.166 INFO:teuthology.orchestra.run.smithi042.stdout:Rotating logs: 2024-04-10T19:05:15.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-04-10T19:05:15.243 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-04-10T19:05:15.245 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2024-04-10T19:05:15.343 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2024-04-10T19:05:15.345 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T19:05:15.345 DEBUG:teuthology.orchestra.run.smithi042:> /usr/sbin/getenforce 2024-04-10T19:05:15.372 INFO:teuthology.orchestra.run.smithi042.stdout:Permissive 2024-04-10T19:05:15.373 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-04-10T19:05:15.404 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-04-10T19:05:15.404 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2024-04-10T19:05:15.431 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2024-04-10T19:05:15.431 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi042.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi062.front.sepia.ceph.com': 'permissive'} 2024-04-10T19:05:15.432 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:05:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:05:15.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:05:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:05:15.491 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:05:15.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:05:15.521 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T19:05:15.521 DEBUG:teuthology.orchestra.run.smithi042:> sudo /usr/sbin/setenforce permissive 2024-04-10T19:05:15.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-04-10T19:05:15.579 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2024-04-10T19:05:15.609 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T19:05:15.618 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T19:05:15.623 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T19:05:15.952 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T19:05:15.966 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T19:05:15.968 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com,smithi045.front.sepia.ceph.com,smithi062.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T19:21:31.478 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964/ansible_failures.yaml 2024-04-10T19:21:31.480 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T19:21:31.682 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=563875e20ce84df7841484c929df07e8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T19:21:31.684 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T19:21:31.692 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T19:21:31.692 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T19:21:31.733 DEBUG:teuthology.orchestra.run.smithi042:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T19:21:31.819 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T19:21:31.888 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T19:21:31.957 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:21:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:21:31.992 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:21:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:21:32.026 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:21:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:21:32.059 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T19:21:32.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T19:21:32.117 INFO:teuthology.task.internal:Duration was 977.110171 seconds 2024-04-10T19:21:32.117 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T19:21:32.150 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T19:21:32.150 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T19:21:32.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T19:21:32.155 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T19:21:32.205 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:21:32.206 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:21:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:21:32.623 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T19:21:32.623 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com 2024-04-10T19:21:32.623 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T19:21:32.651 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-04-10T19:21:32.651 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T19:21:32.679 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-04-10T19:21:32.679 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T19:21:32.708 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T19:21:32.708 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T19:21:32.711 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T19:21:32.722 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T19:21:32.793 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T19:21:32.794 DEBUG:teuthology.orchestra.run.smithi042:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T19:21:32.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T19:21:32.799 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T19:21:32.879 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T19:21:32.887 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T19:21:32.888 DEBUG:teuthology.orchestra.run.smithi042:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T19:21:32.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T19:21:32.945 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T19:21:32.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T19:21:32.982 DEBUG:teuthology.orchestra.run.smithi042:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:21:32.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:21:32.988 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:21:33.010 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = core 2024-04-10T19:21:33.014 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-04-10T19:21:33.041 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-04-10T19:21:33.059 DEBUG:teuthology.orchestra.run.smithi042:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:21:33.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:21:33.096 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:21:33.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:21:33.114 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:21:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:21:33.131 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T19:21:33.139 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T19:21:33.141 DEBUG:teuthology.misc:Transferring archived files from smithi042:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964/remote/smithi042 2024-04-10T19:21:33.142 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T19:21:33.193 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964/remote/smithi045 2024-04-10T19:21:33.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T19:21:33.251 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964/remote/smithi062 2024-04-10T19:21:33.252 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T19:21:33.326 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T19:21:33.326 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T19:21:33.328 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T19:21:33.331 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T19:21:33.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T19:21:33.359 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T19:21:33.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T19:21:33.384 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T19:21:33.384 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T19:21:33.388 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T19:21:33.390 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T19:21:33.404 INFO:teuthology.orchestra.run.smithi042.stdout: 525354 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 19:21 /home/ubuntu/cephtest 2024-04-10T19:21:33.405 INFO:teuthology.orchestra.run.smithi045.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 19:21 /home/ubuntu/cephtest 2024-04-10T19:21:33.408 INFO:teuthology.orchestra.run.smithi062.stdout: 525354 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 19:21 /home/ubuntu/cephtest 2024-04-10T19:21:33.409 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T19:21:33.417 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T19:21:33.485 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T19:21:33.525 DEBUG:teuthology.nuke:shortname: smithi042 2024-04-10T19:21:33.525 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:21:33.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 18:37:09.225074', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEl1i8WX1Ko2umMFieFEYt+gSb/TkAHDhIUCnmpspZGL8k1S1veDgICVGyCDtB+l+z7V61XAYTjmQPgs1aLcud0='} 2024-04-10T19:21:33.586 DEBUG:teuthology.nuke:shortname: smithi045 2024-04-10T19:21:33.586 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:21:33.635 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 18:37:09.222029', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI/n8ijkP3hAhBmUmRxMZ/F4RVDw5jmKZT0xvhN+QkxrG2aFADvtClcqyoXPDgBp8e0QFQ131hSIVaQ9OT1/44c='} 2024-04-10T19:21:33.655 DEBUG:teuthology.nuke:shortname: smithi062 2024-04-10T19:21:33.655 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:21:33.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7650964', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 18:37:09.223617', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF5Nc+nE5t/6ejaWrpuQGqQSNHGMiW0qUe8ueumrm5RCyOAEU/1HG2IRKtLYy8UbbetzIvtDu4qfpv/AQsAWHaw='} 2024-04-10T19:21:33.737 INFO:teuthology.orchestra.console.smithi042:Power off 2024-04-10T19:21:33.737 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T19:21:33.788 INFO:teuthology.orchestra.console.smithi045:Power off 2024-04-10T19:21:33.789 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T19:21:33.813 DEBUG:teuthology.orchestra.console.smithi042:power off output: Chassis Power Control: Down/Off 2024-04-10T19:21:33.814 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:33.843 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-04-10T19:21:33.843 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:33.855 INFO:teuthology.orchestra.console.smithi062:Power off 2024-04-10T19:21:33.856 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T19:21:33.874 DEBUG:teuthology.orchestra.console.smithi042:check power output: Chassis Power is on 2024-04-10T19:21:33.906 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-10T19:21:33.910 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2024-04-10T19:21:33.910 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:33.966 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-10T19:21:37.875 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:37.907 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:37.967 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:37.989 DEBUG:teuthology.orchestra.console.smithi042:check power output: Chassis Power is on 2024-04-10T19:21:38.020 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-10T19:21:38.077 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-10T19:21:41.994 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:42.021 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:42.078 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:42.105 DEBUG:teuthology.orchestra.console.smithi042:check power output: Chassis Power is off 2024-04-10T19:21:42.131 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-10T19:21:42.187 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-10T19:21:42.206 INFO:teuthology.orchestra.console.smithi042:Power off completed 2024-04-10T19:21:42.232 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-04-10T19:21:46.192 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:21:46.302 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2024-04-10T19:21:46.402 INFO:teuthology.orchestra.console.smithi062:Power off completed 2024-04-10T19:21:46.508 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} duration: 977.1101713180542 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=563875e20ce84df7841484c929df07e8 status: dead success: false 2024-04-10T19:21:46.508 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T19:21:46.624 INFO:teuthology.run:DEAD