2024-04-10T17:08:48.234 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T17:08:48.234 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} --verbose --archive /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818 -- /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818/orig.config.yaml 2024-04-10T17:08:48.270 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T17:08:48.434 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818 branch: wip-yuri6-testing-2024-04-02-1310 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} email: yweinste@redhat.com first_in_suite: false job_id: '7650818' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2024-04-02-1310 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: a5074d4516d566e9d8b6aec912f26afd099de101 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a5074d4516d566e9d8b6aec912f26afd099de101 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri6-testing-2024-04-02-1310 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 4372 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2024-04-02-1310 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a5074d4516d566e9d8b6aec912f26afd099de101 targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGw8boHm3DXgShJcLxW3E0yupsTRYTihuBJMI350n23gxTZxt0dNPWTTw2j6rizWGsjrRnOo0pyyGBl5J6Zb5pY= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ//1xRY3LJqypT5Sji5mJjg3IW0hVyqeYFfil0JL0jL4J8io+W72pd96Tx3XG3ZzLBVWSuh0EWwLPWOUblyO7g= smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1Yw7EgYBh9ZJ6u41goChfrHWKtR0m4RuPFEQqhw28K9cCcdrFt67611ELmlUWNRnPWFIY22andMTcGVZSrmqg= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - rados: clients: - client.2 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:17:51 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T17:08:48.434 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa; will attempt to use it 2024-04-10T17:08:48.435 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa/tasks 2024-04-10T17:08:48.435 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T17:08:48.436 INFO:teuthology.task.internal:Saving configuration 2024-04-10T17:08:48.449 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T17:08:48.456 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T17:08:48.482 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 17:03:02.783138', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGw8boHm3DXgShJcLxW3E0yupsTRYTihuBJMI350n23gxTZxt0dNPWTTw2j6rizWGsjrRnOo0pyyGBl5J6Zb5pY='} 2024-04-10T17:08:48.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 17:03:02.784825', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ//1xRY3LJqypT5Sji5mJjg3IW0hVyqeYFfil0JL0jL4J8io+W72pd96Tx3XG3ZzLBVWSuh0EWwLPWOUblyO7g='} 2024-04-10T17:08:48.532 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 17:03:02.786083', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1Yw7EgYBh9ZJ6u41goChfrHWKtR0m4RuPFEQqhw28K9cCcdrFt67611ELmlUWNRnPWFIY22andMTcGVZSrmqg='} 2024-04-10T17:08:48.532 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T17:08:48.538 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T17:08:48.538 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T17:08:48.538 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T17:08:48.538 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T17:08:48.677 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe814263f70>, signals=[15]) 2024-04-10T17:08:48.677 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T17:08:48.683 INFO:teuthology.task.internal:Opening connections... 2024-04-10T17:08:48.683 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-04-10T17:08:48.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:08:48.757 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-04-10T17:08:48.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:08:48.830 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-04-10T17:08:48.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:08:48.901 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T17:08:48.909 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-04-10T17:08:48.931 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-04-10T17:08:48.933 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8" 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8" 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T17:08:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-04-10T17:08:48.995 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T17:08:48.995 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T17:08:48.995 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T17:08:48.995 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-04-10T17:08:49.021 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-04-10T17:08:49.044 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-04-10T17:08:49.044 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="8" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="8" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el8" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-04-10T17:08:49.105 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T17:08:49.106 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T17:08:49.106 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T17:08:49.106 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-04-10T17:08:49.129 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-04-10T17:08:49.151 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-04-10T17:08:49.152 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8" 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T17:08:49.213 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2024-04-10T17:08:49.214 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T17:08:49.214 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T17:08:49.214 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T17:08:49.214 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-04-10T17:08:49.245 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T17:08:49.255 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T17:08:49.260 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T17:08:49.260 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-04-10T17:08:49.263 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-04-10T17:08:49.265 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-04-10T17:08:49.285 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T17:08:49.352 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T17:08:49.352 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-04-10T17:08:49.354 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-04-10T17:08:49.357 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-04-10T17:08:49.420 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T17:08:49.490 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T17:08:49.504 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T17:08:49.505 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T17:08:49.505 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T17:08:49.506 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T17:08:49.506 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T17:08:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:08:49.527 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-04-10T17:08:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:08:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:08:49.529 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-04-10T17:08:49.529 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-04-10T17:08:49.589 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-489.el8.x86_64 2024-04-10T17:08:49.589 INFO:teuthology.task.kernel:Running kernel on smithi131: 4.18.0-489.el8.x86_64 2024-04-10T17:08:49.589 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum install -y kernel 2024-04-10T17:08:49.590 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-489.el8.x86_64 2024-04-10T17:08:49.590 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-489.el8.x86_64 2024-04-10T17:08:49.590 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2024-04-10T17:08:49.591 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-489.el8.x86_64 2024-04-10T17:08:49.592 INFO:teuthology.task.kernel:Running kernel on smithi078: 4.18.0-489.el8.x86_64 2024-04-10T17:08:49.592 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-04-10T17:08:54.866 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - AppStream 9.9 MB/s | 28 MB 00:02 2024-04-10T17:08:55.056 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - AppStream 9.9 MB/s | 28 MB 00:02 2024-04-10T17:08:56.918 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - AppStream 5.8 MB/s | 28 MB 00:04 2024-04-10T17:09:02.597 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - BaseOS 4.0 MB/s | 10 MB 00:02 2024-04-10T17:09:05.289 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras 17 kB/s | 18 kB 00:01 2024-04-10T17:09:50.176 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras common packages 176 B/s | 7.7 kB 00:44 2024-04-10T17:11:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - BaseOS 80 kB/s | 10 MB 02:11 2024-04-10T17:11:16.223 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - BaseOS 77 kB/s | 10 MB 02:16 2024-04-10T17:11:16.288 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras 18 kB/s | 18 kB 00:00 2024-04-10T17:11:18.626 INFO:teuthology.orchestra.run.smithi131.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T17:11:18.627 INFO:teuthology.orchestra.run.smithi131.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T17:11:18.627 INFO:teuthology.orchestra.run.smithi131.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T17:11:18.628 INFO:teuthology.orchestra.run.smithi131.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T17:11:29.674 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 1.5 MB/s | 16 MB 00:11 2024-04-10T17:11:33.390 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 633 kB/s | 24 kB 00:00 2024-04-10T17:11:33.530 INFO:teuthology.orchestra.run.smithi131.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T17:11:36.371 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T17:11:36.371 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T17:11:36.491 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-04-10T17:11:36.492 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-04-10T17:11:36.493 INFO:teuthology.orchestra.run.smithi131.stdout:Install 3 Packages 2024-04-10T17:11:36.493 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T17:11:36.493 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 90 M 2024-04-10T17:11:36.493 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 96 M 2024-04-10T17:11:36.493 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-04-10T17:11:39.066 INFO:teuthology.orchestra.run.smithi131.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 4.5 MB/s | 10 MB 00:02 2024-04-10T17:11:40.425 INFO:teuthology.orchestra.run.smithi131.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 10 MB/s | 36 MB 00:03 2024-04-10T17:11:42.292 INFO:teuthology.orchestra.run.smithi131.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 7.9 MB/s | 44 MB 00:05 2024-04-10T17:11:42.292 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-04-10T17:11:42.292 INFO:teuthology.orchestra.run.smithi131.stdout:Total 16 MB/s | 90 MB 00:05 2024-04-10T17:11:42.978 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-04-10T17:11:43.403 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-04-10T17:11:43.403 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-04-10T17:11:44.643 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-04-10T17:11:44.662 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-04-10T17:11:47.461 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-04-10T17:11:48.031 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:11:51.624 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:11:52.055 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:11:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:11:55.769 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:12:01.163 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras common packages 176 B/s | 7.7 kB 00:44 2024-04-10T17:12:17.725 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:12:17.750 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:12:18.565 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:12:18.565 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:12:18.565 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:12:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:12:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T17:12:19.086 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-04-10T17:12:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T17:12:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T17:12:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T17:12:19.087 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-04-10T17:12:19.243 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:12:19.462 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:12:19.463 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:12:19.463 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T17:12:19.463 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T17:12:19.463 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T17:12:19.463 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T17:12:19.463 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T17:12:19.463 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:12:19.675 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:12:19.675 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:12:19.675 DEBUG:teuthology.orchestra.run.smithi131:> sudo rpm -qi grub2-tools 2024-04-10T17:12:19.708 INFO:teuthology.orchestra.run.smithi131.stdout:Name : grub2-tools 2024-04-10T17:12:19.708 INFO:teuthology.orchestra.run.smithi131.stdout:Epoch : 1 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Version : 2.02 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Release : 129.el8 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: x86_64 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Group : System Environment/Base 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Size : 9265586 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:License : GPLv3+ 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T17:12:19.709 INFO:teuthology.orchestra.run.smithi131.stdout:Relocations : (not relocatable) 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:Packager : CentOS BuildSystem 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:Vendor : CentOS 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:Summary : Support tools for GRUB. 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:Description : 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:hardware devices. 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T17:12:19.710 INFO:teuthology.orchestra.run.smithi131.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T17:12:19.711 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T17:12:19.711 INFO:teuthology.task.kernel:Updating grub on smithi131 to boot 4.18.0-552.el8.x86_64 2024-04-10T17:12:19.711 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T17:12:21.097 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2024-04-10T17:12:23.381 INFO:teuthology.orchestra.run.smithi131.stderr:done 2024-04-10T17:12:23.383 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2024-04-10T17:12:23.423 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.WRRaFmlqRO 2024-04-10T17:12:23.424 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WRRaFmlqRO 2024-04-10T17:12:23.496 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.WRRaFmlqRO 2024-04-10T17:12:23.651 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.WRRaFmlqRO is 6KB 2024-04-10T17:12:23.699 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.WRRaFmlqRO 2024-04-10T17:12:23.716 DEBUG:teuthology.orchestra.run.smithi131:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T17:12:23.788 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T17:12:23.788 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T17:12:23.788 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T17:12:23.789 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T17:12:23.789 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T17:12:23.926 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2024-04-10T17:12:53.929 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T17:12:53.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T17:12:53.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:13:12.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-04-10T17:13:21.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T17:13:21.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:13:29.605 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T17:13:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T17:13:29.606 INFO:teuthology.orchestra.run.smithi078.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T17:13:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T17:13:30.954 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras 137 B/s | 18 kB 02:13 2024-04-10T17:14:15.824 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras common packages 176 B/s | 7.7 kB 00:44 2024-04-10T17:14:21.483 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T17:14:33.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T17:14:33.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:14:33.890 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-04-10T17:14:34.182 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T17:14:34.182 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T17:14:34.182 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-04-10T17:14:34.202 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-552.el8.x86_64 2024-04-10T17:14:34.202 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T17:14:34.202 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T17:14:34.202 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T17:14:35.203 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T17:14:35.204 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T17:14:35.277 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2024-04-10T17:14:35.289 DEBUG:teuthology.parallel:result is None 2024-04-10T17:15:42.067 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:26 2024-04-10T17:15:42.068 INFO:teuthology.orchestra.run.smithi148.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T17:15:42.068 INFO:teuthology.orchestra.run.smithi148.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T17:15:42.068 INFO:teuthology.orchestra.run.smithi148.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T17:15:44.775 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 6.0 MB/s | 16 MB 00:02 2024-04-10T17:15:48.614 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 174 kB/s | 24 kB 00:00 2024-04-10T17:15:48.756 INFO:teuthology.orchestra.run.smithi148.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T17:15:51.586 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T17:15:51.587 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T17:15:51.704 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-10T17:15:51.705 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-10T17:15:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-10T17:15:51.705 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-10T17:15:51.705 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-10T17:15:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T17:15:51.705 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-10T17:15:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T17:15:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T17:15:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T17:15:51.706 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-10T17:15:51.706 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-10T17:15:51.706 INFO:teuthology.orchestra.run.smithi148.stdout:Install 3 Packages 2024-04-10T17:15:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T17:15:51.707 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 90 M 2024-04-10T17:15:51.707 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 96 M 2024-04-10T17:15:51.707 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-10T17:15:53.303 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 116 kB/s | 16 MB 02:23 2024-04-10T17:15:54.115 INFO:teuthology.orchestra.run.smithi148.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 5.1 MB/s | 10 MB 00:02 2024-04-10T17:15:56.399 INFO:teuthology.orchestra.run.smithi148.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 8.4 MB/s | 36 MB 00:04 2024-04-10T17:15:56.782 INFO:teuthology.orchestra.run.smithi148.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 9.3 MB/s | 44 MB 00:04 2024-04-10T17:15:56.782 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-10T17:15:56.783 INFO:teuthology.orchestra.run.smithi148.stdout:Total 18 MB/s | 90 MB 00:05 2024-04-10T17:15:56.994 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 595 kB/s | 24 kB 00:00 2024-04-10T17:15:57.141 INFO:teuthology.orchestra.run.smithi078.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T17:15:57.464 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-10T17:15:57.895 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-10T17:15:57.895 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-10T17:15:59.157 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-10T17:15:59.179 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-10T17:15:59.957 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T17:15:59.958 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T17:16:00.070 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout:Install 3 Packages 2024-04-10T17:16:00.071 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-10T17:16:00.072 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 90 M 2024-04-10T17:16:00.072 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 96 M 2024-04-10T17:16:00.072 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-04-10T17:16:01.993 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-10T17:16:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:16:06.128 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:16:06.491 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:16:09.927 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:16:10.222 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:16:32.059 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:16:32.084 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:16:32.425 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:16:32.425 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:16:32.425 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:16:33.214 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:16:33.214 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T17:16:33.214 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-10T17:16:33.214 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T17:16:33.214 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T17:16:33.214 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T17:16:33.214 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-10T17:16:33.352 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:16:33.574 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:16:33.575 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:16:33.576 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T17:16:33.576 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T17:16:33.576 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T17:16:33.576 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T17:16:33.576 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T17:16:33.576 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:16:33.787 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:16:33.789 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:16:33.789 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2024-04-10T17:16:33.822 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2024-04-10T17:16:33.822 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2024-04-10T17:16:33.822 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.02 2024-04-10T17:16:33.822 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 129.el8 2024-04-10T17:16:33.822 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2024-04-10T17:16:33.822 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Group : System Environment/Base 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 9265586 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Relocations : (not relocatable) 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : CentOS BuildSystem 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T17:16:33.823 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2024-04-10T17:16:33.824 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2024-04-10T17:16:33.824 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T17:16:33.824 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T17:16:33.824 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T17:16:33.824 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T17:16:33.824 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2024-04-10T17:16:33.824 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-10T17:16:33.824 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T17:16:33.825 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T17:16:33.825 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 4.18.0-552.el8.x86_64 2024-04-10T17:16:33.825 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T17:16:35.122 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2024-04-10T17:16:37.259 INFO:teuthology.orchestra.run.smithi148.stderr:done 2024-04-10T17:16:37.260 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-04-10T17:16:37.297 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.imael5CXRd 2024-04-10T17:16:37.297 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.imael5CXRd 2024-04-10T17:16:37.371 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.imael5CXRd 2024-04-10T17:16:37.522 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.imael5CXRd is 6KB 2024-04-10T17:16:37.544 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.imael5CXRd 2024-04-10T17:16:37.562 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T17:16:37.628 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T17:16:37.628 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T17:16:37.628 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T17:16:37.628 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T17:16:37.629 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T17:16:37.791 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2024-04-10T17:17:07.794 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T17:17:07.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-10T17:17:07.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:17:42.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-04-10T17:17:51.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-10T17:17:51.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:18:09.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-04-10T17:18:11.389 INFO:teuthology.orchestra.run.smithi078.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 5.0 MB/s | 10 MB 00:02 2024-04-10T17:18:13.047 INFO:teuthology.orchestra.run.smithi078.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 12 MB/s | 44 MB 00:03 2024-04-10T17:18:14.073 INFO:teuthology.orchestra.run.smithi078.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 7.7 MB/s | 36 MB 00:04 2024-04-10T17:18:14.073 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-04-10T17:18:14.073 INFO:teuthology.orchestra.run.smithi078.stdout:Total 691 kB/s | 90 MB 02:14 2024-04-10T17:18:14.761 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-04-10T17:18:15.187 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-04-10T17:18:15.187 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-04-10T17:18:16.433 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-04-10T17:18:16.468 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-04-10T17:18:19.253 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-04-10T17:18:19.793 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:18:21.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-10T17:18:21.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:18:23.340 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:18:23.768 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:18:25.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-04-10T17:18:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:18:27.508 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:18:40.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-10T17:18:40.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:18:40.551 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-04-10T17:18:40.811 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-10T17:18:40.811 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T17:18:40.811 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-04-10T17:18:40.830 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-552.el8.x86_64 2024-04-10T17:18:40.830 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T17:18:40.830 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T17:18:40.830 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T17:18:41.831 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T17:18:41.831 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T17:18:41.911 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2024-04-10T17:18:41.928 DEBUG:teuthology.parallel:result is None 2024-04-10T17:18:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:18:49.459 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:18:49.793 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:18:49.794 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T17:18:49.794 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T17:18:50.241 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T17:18:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-10T17:18:50.242 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-04-10T17:18:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T17:18:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T17:18:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-10T17:18:50.242 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-04-10T17:18:50.432 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:18:50.651 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:18:50.651 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:18:50.652 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T17:18:50.652 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T17:18:50.652 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T17:18:50.652 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T17:18:50.652 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T17:18:50.652 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T17:18:50.862 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T17:18:50.862 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T17:18:50.862 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-04-10T17:18:50.896 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-04-10T17:18:50.896 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-04-10T17:18:50.896 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.02 2024-04-10T17:18:50.896 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 129.el8 2024-04-10T17:18:50.896 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-04-10T17:18:50.896 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T17:18:50.896 INFO:teuthology.orchestra.run.smithi078.stdout:Group : System Environment/Base 2024-04-10T17:18:50.896 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 9265586 2024-04-10T17:18:50.896 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:Relocations : (not relocatable) 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : CentOS BuildSystem 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T17:18:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-04-10T17:18:50.898 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-10T17:18:50.898 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T17:18:50.898 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T17:18:50.898 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 4.18.0-552.el8.x86_64 2024-04-10T17:18:50.899 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T17:18:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-04-10T17:18:55.270 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-04-10T17:18:55.272 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-04-10T17:18:55.295 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.KpC6MJYgWg 2024-04-10T17:18:55.295 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KpC6MJYgWg 2024-04-10T17:18:55.368 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.KpC6MJYgWg 2024-04-10T17:18:55.513 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.KpC6MJYgWg is 6KB 2024-04-10T17:18:55.560 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.KpC6MJYgWg 2024-04-10T17:18:55.577 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T17:18:55.650 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T17:18:55.651 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T17:18:55.651 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T17:18:55.651 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T17:18:55.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T17:18:55.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-04-10T17:19:25.799 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T17:19:25.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-10T17:19:25.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:20:00.546 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-04-10T17:20:09.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-10T17:20:09.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:20:28.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-04-10T17:20:40.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-10T17:20:40.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T17:20:40.666 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-04-10T17:20:40.939 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-10T17:20:40.939 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T17:20:40.939 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-04-10T17:20:40.956 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-552.el8.x86_64 2024-04-10T17:20:40.956 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T17:20:40.956 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T17:20:40.956 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T17:20:41.957 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T17:20:41.957 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T17:20:42.177 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-04-10T17:20:42.207 DEBUG:teuthology.parallel:result is None 2024-04-10T17:20:42.208 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T17:20:42.215 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T17:20:42.216 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T17:20:42.229 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T17:20:42.232 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T17:20:42.290 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T17:20:42.297 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T17:20:42.308 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T17:20:42.308 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T17:20:42.321 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T17:20:42.323 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T17:20:42.463 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T17:20:42.472 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T17:20:42.472 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T17:20:42.492 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T17:20:42.494 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T17:20:42.538 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:20:42.545 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:20:42.550 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:20:42.557 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:20:42.949 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:20:43.057 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T17:20:43.069 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T17:20:43.075 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T17:20:43.075 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T17:20:43.115 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T17:20:43.118 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T17:20:43.602 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T17:20:43.612 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T17:20:43.612 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T17:20:43.657 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T17:20:43.659 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T17:20:43.840 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:20:43.973 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:20:44.104 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:20:44.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:20:44.240 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-10T17:20:44.240 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T17:20:44.306 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:20:44.419 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:20:44.460 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:20:44.626 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:20:44.655 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-10T17:20:44.655 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T17:20:44.724 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:20:44.846 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T17:20:44.893 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:20:45.001 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T17:20:45.031 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-10T17:20:45.031 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T17:20:45.100 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-04-10T17:20:45.103 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-04-10T17:20:45.105 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-04-10T17:20:45.157 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:20:45.195 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:20:45.213 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:20:45.621 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T17:20:45.628 INFO:teuthology.task.internal:Starting timer... 2024-04-10T17:20:45.629 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T17:20:45.639 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T17:20:45.646 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T17:20:45.647 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-04-10T17:20:45.726 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-04-10T17:20:45.728 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2024-04-10T17:20:45.812 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2024-04-10T17:20:45.814 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2024-04-10T17:20:45.879 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2024-04-10T17:20:45.880 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T17:20:45.880 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-04-10T17:20:45.909 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-04-10T17:20:45.910 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2024-04-10T17:20:45.938 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2024-04-10T17:20:45.939 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2024-04-10T17:20:46.009 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2024-04-10T17:20:46.010 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi131.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive'} 2024-04-10T17:20:46.010 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:20:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:20:46.036 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:20:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:20:46.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:20:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:20:46.093 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T17:20:46.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-04-10T17:20:46.119 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2024-04-10T17:20:46.155 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2024-04-10T17:20:46.183 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T17:20:46.194 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T17:20:46.199 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T17:20:46.734 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T17:20:46.747 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T17:20:46.748 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi131.front.sepia.ceph.com,smithi148.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T17:39:19.310 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818/ansible_failures.yaml 2024-04-10T17:39:19.314 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T17:39:19.521 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8e289e7eef024abdbd4a78824af59184 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T17:39:19.523 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T17:39:19.530 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T17:39:19.531 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T17:39:19.553 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T17:39:19.622 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T17:39:19.685 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T17:39:19.749 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:39:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:39:19.784 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:39:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:39:19.816 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T17:39:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:39:19.848 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T17:39:19.855 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T17:39:19.863 INFO:teuthology.task.internal:Duration was 1114.234233 seconds 2024-04-10T17:39:19.863 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T17:39:19.872 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T17:39:19.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T17:39:19.875 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T17:39:19.877 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T17:39:19.924 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:39:19.927 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:39:19.936 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T17:39:20.372 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T17:39:20.373 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-04-10T17:39:20.373 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T17:39:20.400 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-04-10T17:39:20.400 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T17:39:20.426 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-04-10T17:39:20.426 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T17:39:20.450 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T17:39:20.451 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T17:39:20.453 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T17:39:20.469 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T17:39:20.533 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T17:39:20.533 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T17:39:20.535 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T17:39:20.553 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T17:39:20.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T17:39:20.652 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T17:39:20.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T17:39:20.684 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T17:39:20.712 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T17:39:20.739 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T17:39:20.748 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:39:20.750 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:39:20.755 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:39:20.777 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-04-10T17:39:20.780 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-04-10T17:39:20.806 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-04-10T17:39:20.828 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:39:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:39:20.854 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:39:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:39:20.871 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T17:39:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T17:39:20.888 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T17:39:20.896 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T17:39:20.898 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818/remote/smithi078 2024-04-10T17:39:20.899 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T17:39:20.949 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818/remote/smithi131 2024-04-10T17:39:20.950 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T17:39:21.006 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818/remote/smithi148 2024-04-10T17:39:21.007 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T17:39:21.058 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T17:39:21.058 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T17:39:21.060 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T17:39:21.062 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T17:39:21.110 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T17:39:21.120 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T17:39:21.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T17:39:21.128 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T17:39:21.128 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T17:39:21.130 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T17:39:21.132 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T17:39:21.147 INFO:teuthology.orchestra.run.smithi078.stdout: 526074 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 17:39 /home/ubuntu/cephtest 2024-04-10T17:39:21.147 INFO:teuthology.orchestra.run.smithi131.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 17:39 /home/ubuntu/cephtest 2024-04-10T17:39:21.167 INFO:teuthology.orchestra.run.smithi148.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 17:39 /home/ubuntu/cephtest 2024-04-10T17:39:21.168 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T17:39:21.177 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T17:39:21.195 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T17:39:21.222 DEBUG:teuthology.nuke:shortname: smithi078 2024-04-10T17:39:21.223 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T17:39:21.248 DEBUG:teuthology.nuke:shortname: smithi131 2024-04-10T17:39:21.248 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T17:39:21.253 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 17:03:02.783138', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGw8boHm3DXgShJcLxW3E0yupsTRYTihuBJMI350n23gxTZxt0dNPWTTw2j6rizWGsjrRnOo0pyyGBl5J6Zb5pY='} 2024-04-10T17:39:21.268 DEBUG:teuthology.nuke:shortname: smithi148 2024-04-10T17:39:21.269 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T17:39:21.274 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 17:03:02.784825', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ//1xRY3LJqypT5Sji5mJjg3IW0hVyqeYFfil0JL0jL4J8io+W72pd96Tx3XG3ZzLBVWSuh0EWwLPWOUblyO7g='} 2024-04-10T17:39:21.289 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650818', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 17:03:02.786083', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1Yw7EgYBh9ZJ6u41goChfrHWKtR0m4RuPFEQqhw28K9cCcdrFt67611ELmlUWNRnPWFIY22andMTcGVZSrmqg='} 2024-04-10T17:39:21.306 INFO:teuthology.orchestra.console.smithi078:Power off 2024-04-10T17:39:21.306 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T17:39:21.334 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2024-04-10T17:39:21.334 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:21.350 INFO:teuthology.orchestra.console.smithi148:Power off 2024-04-10T17:39:21.351 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T17:39:21.363 INFO:teuthology.orchestra.console.smithi131:Power off 2024-04-10T17:39:21.364 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T17:39:21.373 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2024-04-10T17:39:21.374 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:21.381 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-04-10T17:39:21.387 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-04-10T17:39:21.387 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:21.396 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-04-10T17:39:21.408 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-04-10T17:39:25.382 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:25.397 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:25.409 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:25.493 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-04-10T17:39:25.506 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-04-10T17:39:25.517 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-04-10T17:39:29.496 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:29.508 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:29.518 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:29.606 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-04-10T17:39:29.616 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-04-10T17:39:29.626 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-04-10T17:39:29.727 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-04-10T17:39:33.611 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:33.621 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T17:39:33.723 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2024-04-10T17:39:33.730 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2024-04-10T17:39:33.824 INFO:teuthology.orchestra.console.smithi078:Power off completed 2024-04-10T17:39:33.831 INFO:teuthology.orchestra.console.smithi148:Power off completed 2024-04-10T17:39:33.937 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} duration: 1114.2342331409454 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8e289e7eef024abdbd4a78824af59184 status: dead success: false 2024-04-10T17:39:33.937 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T17:39:34.028 INFO:teuthology.run:DEAD