2024-04-10T19:28:59.119 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T19:28:59.119 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055 --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055/orig.config.yaml 2024-04-10T19:28:59.158 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T19:28:59.246 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055 branch: wip-yuri6-testing-2024-04-02-1310 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} email: yweinste@redhat.com first_in_suite: false job_id: '7651055' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'rbd object class functional tests ' name: yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2024-04-02-1310 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: a5074d4516d566e9d8b6aec912f26afd099de101 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a5074d4516d566e9d8b6aec912f26afd099de101 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri6-testing-2024-04-02-1310 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 4372 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2024-04-02-1310 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a5074d4516d566e9d8b6aec912f26afd099de101 targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHCpyZYztk7Ra0Q6nOd2KbAouMs0g2G55CBKFPs3IInn4t5n0iq035YzK7HdnAmAqjoECvOGTV4i3m79DyY1PfU= smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPjJ4nhRr+25DGl/pRQUcmtZCaiEcVmIuqjFZf093bRN6+vT+mut3s5Sxb6QgNyn/gSBPPeP6Qyx+DWqRNcvS6U= smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIWlv5zdgEbJ5eclJjXBh/QTJjTB/pHlM+ouDzkvuMCNObNjzzCDOIhfhTchCxHQyoadh39PUCZXHXd1jkQkQlk= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: reef exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - exec: client.2: - ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:17:51 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T19:28:59.246 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa; will attempt to use it 2024-04-10T19:28:59.247 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa/tasks 2024-04-10T19:28:59.247 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T19:28:59.248 INFO:teuthology.task.internal:Saving configuration 2024-04-10T19:28:59.263 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T19:28:59.269 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:28:59.303 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:23:59.311085', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHCpyZYztk7Ra0Q6nOd2KbAouMs0g2G55CBKFPs3IInn4t5n0iq035YzK7HdnAmAqjoECvOGTV4i3m79DyY1PfU='} 2024-04-10T19:28:59.339 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:23:59.312755', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPjJ4nhRr+25DGl/pRQUcmtZCaiEcVmIuqjFZf093bRN6+vT+mut3s5Sxb6QgNyn/gSBPPeP6Qyx+DWqRNcvS6U='} 2024-04-10T19:28:59.374 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:23:59.309280', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIWlv5zdgEbJ5eclJjXBh/QTJjTB/pHlM+ouDzkvuMCNObNjzzCDOIhfhTchCxHQyoadh39PUCZXHXd1jkQkQlk='} 2024-04-10T19:28:59.374 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T19:28:59.380 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T19:28:59.380 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T19:28:59.380 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T19:28:59.380 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T19:28:59.525 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdeb5d0f160>, signals=[15]) 2024-04-10T19:28:59.525 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T19:28:59.532 INFO:teuthology.task.internal:Opening connections... 2024-04-10T19:28:59.532 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-04-10T19:28:59.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:28:59.634 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-04-10T19:28:59.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:28:59.721 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-04-10T19:28:59.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:28:59.811 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T19:28:59.818 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-04-10T19:28:59.840 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-04-10T19:28:59.841 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="8" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="8" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el8" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-04-10T19:28:59.902 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T19:28:59.903 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T19:28:59.903 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T19:28:59.903 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-04-10T19:28:59.930 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-04-10T19:28:59.952 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-04-10T19:28:59.953 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-04-10T19:29:00.014 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-04-10T19:29:00.014 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8" 2024-04-10T19:29:00.014 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-04-10T19:29:00.014 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-04-10T19:29:00.014 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8" 2024-04-10T19:29:00.014 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2024-04-10T19:29:00.015 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T19:29:00.015 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-04-10T19:29:00.015 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T19:29:00.015 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-04-10T19:29:00.015 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T19:29:00.015 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T19:29:00.015 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T19:29:00.015 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-04-10T19:29:00.047 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-04-10T19:29:00.068 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-04-10T19:29:00.068 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-04-10T19:29:00.129 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="8" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="8" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el8" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T19:29:00.130 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T19:29:00.131 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-04-10T19:29:00.163 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T19:29:00.173 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T19:29:00.243 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T19:29:00.244 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-04-10T19:29:00.246 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-04-10T19:29:00.249 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-04-10T19:29:00.269 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T19:29:00.281 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T19:29:00.281 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-04-10T19:29:00.307 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-04-10T19:29:00.311 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-04-10T19:29:00.340 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T19:29:00.453 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T19:29:00.493 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T19:29:00.493 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T19:29:00.493 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T19:29:00.494 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T19:29:00.494 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T19:29:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:29:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:29:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:29:00.516 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-10T19:29:00.517 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-10T19:29:00.517 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-04-10T19:29:00.577 INFO:teuthology.orchestra.run.smithi117.stdout:4.18.0-489.el8.x86_64 2024-04-10T19:29:00.578 INFO:teuthology.task.kernel:Running kernel on smithi117: 4.18.0-489.el8.x86_64 2024-04-10T19:29:00.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-04-10T19:29:00.579 INFO:teuthology.orchestra.run.smithi028.stdout:4.18.0-489.el8.x86_64 2024-04-10T19:29:00.579 INFO:teuthology.orchestra.run.smithi040.stdout:4.18.0-489.el8.x86_64 2024-04-10T19:29:00.580 INFO:teuthology.task.kernel:Running kernel on smithi028: 4.18.0-489.el8.x86_64 2024-04-10T19:29:00.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-04-10T19:29:00.580 INFO:teuthology.task.kernel:Running kernel on smithi040: 4.18.0-489.el8.x86_64 2024-04-10T19:29:00.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2024-04-10T19:29:06.380 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 8 - AppStream 8.5 MB/s | 28 MB 00:03 2024-04-10T19:31:16.021 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - AppStream 215 kB/s | 28 MB 02:12 2024-04-10T19:31:17.512 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 8 - AppStream 213 kB/s | 28 MB 02:14 2024-04-10T19:31:21.563 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 8 - BaseOS 81 kB/s | 10 MB 02:10 2024-04-10T19:31:22.904 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - BaseOS 7.2 MB/s | 10 MB 00:01 2024-04-10T19:31:24.197 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 8 - BaseOS 7.5 MB/s | 10 MB 00:01 2024-04-10T19:32:07.484 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras 425 B/s | 18 kB 00:42 2024-04-10T19:32:07.496 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 8 - Extras 411 B/s | 18 kB 00:44 2024-04-10T19:32:52.013 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 8 - Extras common packages 177 B/s | 7.7 kB 00:44 2024-04-10T19:34:18.056 INFO:teuthology.orchestra.run.smithi117.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:25 2024-04-10T19:34:18.057 INFO:teuthology.orchestra.run.smithi117.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T19:34:18.057 INFO:teuthology.orchestra.run.smithi117.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T19:34:18.057 INFO:teuthology.orchestra.run.smithi117.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T19:34:21.621 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 8 - Extras 104 B/s | 18 kB 02:55 2024-04-10T19:34:21.657 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 16 MB 00:03 2024-04-10T19:34:25.421 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 528 kB/s | 24 kB 00:00 2024-04-10T19:34:25.563 INFO:teuthology.orchestra.run.smithi117.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:34:28.393 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:34:28.393 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:34:28.498 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-04-10T19:34:28.499 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-04-10T19:34:28.499 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-04-10T19:34:28.500 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-04-10T19:34:28.500 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-04-10T19:34:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:34:28.500 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-04-10T19:34:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:34:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:34:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-10T19:34:28.500 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-04-10T19:34:28.500 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-04-10T19:34:28.501 INFO:teuthology.orchestra.run.smithi117.stdout:Install 3 Packages 2024-04-10T19:34:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-10T19:34:28.501 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 90 M 2024-04-10T19:34:28.501 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 96 M 2024-04-10T19:34:28.501 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-04-10T19:35:02.964 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras common packages 45 B/s | 7.7 kB 02:55 2024-04-10T19:35:03.092 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 8 - Extras common packages 191 B/s | 7.7 kB 00:41 2024-04-10T19:36:32.351 INFO:teuthology.orchestra.run.smithi028.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:29 2024-04-10T19:36:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T19:36:32.352 INFO:teuthology.orchestra.run.smithi028.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T19:36:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T19:36:32.422 INFO:teuthology.orchestra.run.smithi040.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:29 2024-04-10T19:36:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T19:36:32.423 INFO:teuthology.orchestra.run.smithi040.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T19:36:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T19:36:34.009 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 16 MB 00:01 2024-04-10T19:36:34.633 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 16 MB 00:02 2024-04-10T19:36:37.943 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 590 kB/s | 24 kB 00:00 2024-04-10T19:36:38.094 INFO:teuthology.orchestra.run.smithi028.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:36:38.623 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 380 kB/s | 24 kB 00:00 2024-04-10T19:36:38.773 INFO:teuthology.orchestra.run.smithi040.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T19:36:40.987 INFO:teuthology.orchestra.run.smithi117.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 8.7 MB/s | 10 MB 00:01 2024-04-10T19:36:41.106 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:36:41.106 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:36:41.228 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-10T19:36:41.229 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-10T19:36:41.229 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-10T19:36:41.229 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout:Install 3 Packages 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 90 M 2024-04-10T19:36:41.230 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 96 M 2024-04-10T19:36:41.231 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-10T19:36:41.795 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T19:36:41.795 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T19:36:41.923 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-10T19:36:41.924 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-10T19:36:41.924 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-10T19:36:41.924 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-10T19:36:41.924 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-10T19:36:41.924 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T19:36:41.924 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-10T19:36:41.925 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T19:36:41.925 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T19:36:41.925 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-10T19:36:41.925 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-10T19:36:41.925 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-10T19:36:41.925 INFO:teuthology.orchestra.run.smithi040.stdout:Install 3 Packages 2024-04-10T19:36:41.925 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-10T19:36:41.926 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 90 M 2024-04-10T19:36:41.926 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 96 M 2024-04-10T19:36:41.926 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-10T19:36:42.063 INFO:teuthology.orchestra.run.smithi117.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-04-10T19:36:42.545 INFO:teuthology.orchestra.run.smithi117.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 16 MB/s | 44 MB 00:02 2024-04-10T19:36:42.546 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:36:42.546 INFO:teuthology.orchestra.run.smithi117.stdout:Total 691 kB/s | 90 MB 02:14 2024-04-10T19:36:43.230 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-04-10T19:36:43.274 INFO:teuthology.orchestra.run.smithi040.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 8.9 MB/s | 10 MB 00:01 2024-04-10T19:36:43.650 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-04-10T19:36:43.650 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-04-10T19:36:44.399 INFO:teuthology.orchestra.run.smithi040.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-04-10T19:36:44.922 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-04-10T19:36:44.951 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-04-10T19:36:45.006 INFO:teuthology.orchestra.run.smithi040.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 15 MB/s | 44 MB 00:02 2024-04-10T19:36:45.007 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:36:45.007 INFO:teuthology.orchestra.run.smithi040.stdout:Total 29 MB/s | 90 MB 00:03 2024-04-10T19:36:45.727 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-10T19:36:46.207 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-10T19:36:46.207 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-10T19:36:47.530 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-10T19:36:47.558 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-10T19:36:47.725 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-04-10T19:36:48.234 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:36:50.479 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-10T19:36:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:36:51.789 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:36:52.200 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:36:54.823 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:36:55.218 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:36:55.651 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:36:55.956 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:36:58.910 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:36:59.231 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:37:17.873 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:37:17.897 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:37:18.246 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:37:18.247 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:37:18.247 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:37:18.754 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:37:18.754 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-10T19:37:18.754 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-04-10T19:37:18.755 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T19:37:18.755 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T19:37:18.755 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-10T19:37:18.755 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-04-10T19:37:18.899 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:37:19.120 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:37:19.121 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:37:19.121 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T19:37:19.121 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T19:37:19.121 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T19:37:19.121 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T19:37:19.121 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T19:37:19.122 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:37:19.332 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:37:19.332 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:37:19.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-04-10T19:37:19.366 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-04-10T19:37:19.367 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-04-10T19:37:19.367 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.02 2024-04-10T19:37:19.367 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 129.el8 2024-04-10T19:37:19.367 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-04-10T19:37:19.367 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T19:37:19.367 INFO:teuthology.orchestra.run.smithi117.stdout:Group : System Environment/Base 2024-04-10T19:37:19.367 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 9265586 2024-04-10T19:37:19.367 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:Relocations : (not relocatable) 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : CentOS BuildSystem 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-04-10T19:37:19.368 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-04-10T19:37:19.369 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-10T19:37:19.369 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T19:37:19.369 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T19:37:19.369 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T19:37:19.369 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-04-10T19:37:19.369 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-10T19:37:19.369 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T19:37:19.370 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T19:37:19.370 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 4.18.0-552.el8.x86_64 2024-04-10T19:37:19.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T19:37:21.019 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-04-10T19:37:22.752 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:37:22.777 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:37:22.857 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-04-10T19:37:22.858 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-04-10T19:37:22.892 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.TgnzB3gt3n 2024-04-10T19:37:22.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TgnzB3gt3n 2024-04-10T19:37:22.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.TgnzB3gt3n 2024-04-10T19:37:23.116 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.TgnzB3gt3n is 6KB 2024-04-10T19:37:23.164 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.TgnzB3gt3n 2024-04-10T19:37:23.181 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T19:37:23.247 INFO:teuthology.orchestra.run.smithi117.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T19:37:23.247 INFO:teuthology.orchestra.run.smithi117.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T19:37:23.247 INFO:teuthology.orchestra.run.smithi117.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T19:37:23.247 INFO:teuthology.orchestra.run.smithi117.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T19:37:23.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T19:37:23.279 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:37:23.279 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:37:23.279 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:37:23.410 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-04-10T19:37:23.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:37:23.876 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-10T19:37:23.876 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-10T19:37:23.876 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T19:37:23.877 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T19:37:23.877 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-10T19:37:23.877 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-10T19:37:24.068 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:37:24.295 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:37:24.296 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:37:24.296 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T19:37:24.296 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T19:37:24.296 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T19:37:24.297 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T19:37:24.297 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T19:37:24.297 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:37:24.518 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:37:24.518 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:37:24.519 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2024-04-10T19:37:24.552 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2024-04-10T19:37:24.552 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2024-04-10T19:37:24.552 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.02 2024-04-10T19:37:24.552 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 129.el8 2024-04-10T19:37:24.552 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Group : System Environment/Base 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 9265586 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Relocations : (not relocatable) 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : CentOS BuildSystem 2024-04-10T19:37:24.553 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-10T19:37:24.554 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T19:37:24.555 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T19:37:24.555 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 4.18.0-552.el8.x86_64 2024-04-10T19:37:24.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T19:37:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-04-10T19:37:27.923 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-04-10T19:37:27.931 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-04-10T19:37:27.960 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.mUe3s5DOTh 2024-04-10T19:37:27.960 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mUe3s5DOTh 2024-04-10T19:37:28.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.mUe3s5DOTh 2024-04-10T19:37:28.153 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.mUe3s5DOTh is 6KB 2024-04-10T19:37:28.203 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.mUe3s5DOTh 2024-04-10T19:37:28.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T19:37:28.302 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T19:37:28.302 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T19:37:28.302 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T19:37:28.303 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T19:37:28.304 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T19:37:28.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2024-04-10T19:37:53.413 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T19:37:53.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-04-10T19:37:53.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:37:58.472 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T19:37:58.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-10T19:37:58.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:38:11.874 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-04-10T19:38:16.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-04-10T19:38:20.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-04-10T19:38:20.877 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:38:23.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-04-10T19:38:25.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-10T19:38:25.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:38:35.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-04-10T19:38:35.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:38:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 8.2 MB/s | 10 MB 00:01 2024-04-10T19:38:54.972 INFO:teuthology.orchestra.run.smithi028.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-04-10T19:38:55.522 INFO:teuthology.orchestra.run.smithi028.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 15 MB/s | 44 MB 00:02 2024-04-10T19:38:55.522 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-10T19:38:55.522 INFO:teuthology.orchestra.run.smithi028.stdout:Total 690 kB/s | 90 MB 02:14 2024-04-10T19:38:56.236 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-10T19:38:56.696 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-10T19:38:56.696 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-10T19:38:58.011 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-10T19:38:58.034 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-10T19:39:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-10T19:39:01.560 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:39:05.280 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:39:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:39:07.817 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-04-10T19:39:08.095 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-04-10T19:39:08.095 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T19:39:08.095 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-04-10T19:39:08.153 INFO:teuthology.orchestra.run.smithi117.stdout:4.18.0-552.el8.x86_64 2024-04-10T19:39:08.154 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T19:39:08.154 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T19:39:08.154 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T19:39:09.155 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T19:39:09.155 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T19:39:09.246 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-04-10T19:39:09.263 DEBUG:teuthology.parallel:result is None 2024-04-10T19:39:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:39:09.596 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:39:26.006 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T19:39:33.122 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:39:33.146 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:39:33.544 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:39:33.544 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T19:39:33.545 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T19:39:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T19:39:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-10T19:39:34.182 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-10T19:39:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T19:39:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T19:39:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-10T19:39:34.182 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-10T19:39:34.732 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:39:34.960 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:39:34.961 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:39:34.961 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T19:39:34.961 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T19:39:34.961 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T19:39:34.961 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T19:39:34.961 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T19:39:34.961 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T19:39:35.182 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T19:39:35.182 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T19:39:35.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-04-10T19:39:35.216 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-04-10T19:39:35.217 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-04-10T19:39:35.217 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.02 2024-04-10T19:39:35.217 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 129.el8 2024-04-10T19:39:35.217 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-04-10T19:39:35.217 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T19:39:35.217 INFO:teuthology.orchestra.run.smithi028.stdout:Group : System Environment/Base 2024-04-10T19:39:35.217 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 9265586 2024-04-10T19:39:35.217 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Relocations : (not relocatable) 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : CentOS BuildSystem 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T19:39:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T19:39:35.219 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T19:39:35.219 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-04-10T19:39:35.219 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-10T19:39:35.219 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T19:39:35.220 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T19:39:35.220 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 4.18.0-552.el8.x86_64 2024-04-10T19:39:35.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T19:39:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-04-10T19:39:38.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-10T19:39:38.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:39:38.368 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-10T19:39:38.650 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-04-10T19:39:38.651 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-04-10T19:39:38.684 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.nbUlL0YiN7 2024-04-10T19:39:38.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nbUlL0YiN7 2024-04-10T19:39:38.693 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-10T19:39:38.693 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T19:39:38.693 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-10T19:39:38.714 INFO:teuthology.orchestra.run.smithi040.stdout:4.18.0-552.el8.x86_64 2024-04-10T19:39:38.714 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T19:39:38.714 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T19:39:38.714 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T19:39:38.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.nbUlL0YiN7 2024-04-10T19:39:38.892 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.nbUlL0YiN7 is 6KB 2024-04-10T19:39:38.941 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.nbUlL0YiN7 2024-04-10T19:39:38.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T19:39:39.038 INFO:teuthology.orchestra.run.smithi028.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T19:39:39.038 INFO:teuthology.orchestra.run.smithi028.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T19:39:39.038 INFO:teuthology.orchestra.run.smithi028.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T19:39:39.039 INFO:teuthology.orchestra.run.smithi028.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T19:39:39.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T19:39:39.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-04-10T19:39:39.715 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T19:39:39.715 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T19:39:39.813 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-04-10T19:39:39.825 DEBUG:teuthology.parallel:result is None 2024-04-10T19:40:09.187 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T19:40:09.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-10T19:40:09.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:40:27.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-04-10T19:40:36.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-10T19:40:36.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:40:54.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-04-10T19:41:06.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-10T19:41:06.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T19:41:07.384 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-10T19:41:07.655 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-10T19:41:07.655 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T19:41:07.656 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-10T19:41:07.673 INFO:teuthology.orchestra.run.smithi028.stdout:4.18.0-552.el8.x86_64 2024-04-10T19:41:07.673 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T19:41:07.673 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T19:41:07.673 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T19:41:08.674 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T19:41:08.674 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T19:41:08.748 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-04-10T19:41:08.765 DEBUG:teuthology.parallel:result is None 2024-04-10T19:41:08.766 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T19:41:08.774 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T19:41:08.774 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T19:41:08.777 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T19:41:08.780 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T19:41:08.803 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T19:41:08.861 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T19:41:08.915 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T19:41:08.916 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T19:41:08.918 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T19:41:08.920 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T19:41:08.962 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T19:41:08.969 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T19:41:08.969 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T19:41:08.992 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T19:41:08.994 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T19:41:09.035 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:41:09.041 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:41:09.046 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:41:09.052 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:41:09.053 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:41:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T19:41:09.065 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T19:41:09.071 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T19:41:09.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T19:41:09.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T19:41:09.090 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T19:41:09.120 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T19:41:09.129 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T19:41:09.129 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T19:41:09.140 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T19:41:09.161 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T19:41:09.180 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:41:09.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:41:09.286 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:41:09.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:41:09.410 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-10T19:41:09.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T19:41:09.474 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:41:09.533 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:41:09.583 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:41:09.640 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:41:09.674 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-10T19:41:09.674 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T19:41:09.743 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:41:09.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T19:41:09.860 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:41:09.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T19:41:09.950 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-10T19:41:09.950 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T19:41:10.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-04-10T19:41:10.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-04-10T19:41:10.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-04-10T19:41:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:41:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:41:10.126 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:41:10.505 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T19:41:10.513 INFO:teuthology.task.internal:Starting timer... 2024-04-10T19:41:10.513 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T19:41:10.569 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T19:41:10.617 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T19:41:10.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-04-10T19:41:10.715 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-04-10T19:41:10.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-04-10T19:41:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-04-10T19:41:10.808 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-04-10T19:41:10.906 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-04-10T19:41:10.908 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T19:41:10.908 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-04-10T19:41:10.942 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-04-10T19:41:10.942 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-04-10T19:41:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-04-10T19:41:10.967 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-04-10T19:41:10.988 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-04-10T19:41:10.988 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi117.front.sepia.ceph.com': 'permissive'} 2024-04-10T19:41:10.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:41:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:41:11.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:41:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:41:11.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:41:11.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:41:11.072 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T19:41:11.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-04-10T19:41:11.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-04-10T19:41:11.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-04-10T19:41:11.156 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T19:41:11.172 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T19:41:11.177 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T19:41:11.325 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T19:41:11.340 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T19:41:11.341 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi040.front.sepia.ceph.com,smithi117.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T19:59:18.373 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055/ansible_failures.yaml 2024-04-10T19:59:18.375 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T19:59:18.582 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=537570e869f84fdfad8514d55867f48f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T19:59:18.584 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T19:59:18.593 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T19:59:18.594 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T19:59:18.627 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T19:59:18.710 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T19:59:18.776 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T19:59:18.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:59:18.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:59:18.877 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:59:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:59:18.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T19:59:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:59:18.941 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T19:59:18.950 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T19:59:18.997 INFO:teuthology.task.internal:Duration was 1088.483968 seconds 2024-04-10T19:59:18.998 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T19:59:19.022 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T19:59:19.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T19:59:19.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T19:59:19.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T19:59:19.075 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:59:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:59:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T19:59:19.482 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T19:59:19.482 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-04-10T19:59:19.482 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T19:59:19.510 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-04-10T19:59:19.510 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T19:59:19.538 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-04-10T19:59:19.538 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T19:59:19.565 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T19:59:19.565 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T19:59:19.568 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T19:59:19.580 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T19:59:19.649 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T19:59:19.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T19:59:19.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T19:59:19.654 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T19:59:19.759 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T19:59:19.769 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T19:59:19.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T19:59:19.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T19:59:19.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T19:59:19.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T19:59:19.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:59:19.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:59:19.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:59:19.893 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-04-10T19:59:19.898 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-04-10T19:59:19.923 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-04-10T19:59:19.947 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:59:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:59:19.971 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:59:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:59:19.989 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T19:59:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T19:59:20.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T19:59:20.016 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T19:59:20.017 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055/remote/smithi028 2024-04-10T19:59:20.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T19:59:20.069 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055/remote/smithi040 2024-04-10T19:59:20.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T19:59:20.118 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055/remote/smithi117 2024-04-10T19:59:20.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T19:59:20.165 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T19:59:20.166 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T19:59:20.168 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T19:59:20.170 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T19:59:20.217 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T19:59:20.228 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T19:59:20.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T19:59:20.293 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T19:59:20.293 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T19:59:20.295 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T19:59:20.299 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T19:59:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: 525703 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 19:59 /home/ubuntu/cephtest 2024-04-10T19:59:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 19:59 /home/ubuntu/cephtest 2024-04-10T19:59:20.316 INFO:teuthology.orchestra.run.smithi117.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 19:59 /home/ubuntu/cephtest 2024-04-10T19:59:20.318 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T19:59:20.327 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T19:59:20.363 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T19:59:20.405 DEBUG:teuthology.nuke:shortname: smithi028 2024-04-10T19:59:20.405 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:59:20.452 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:23:59.311085', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHCpyZYztk7Ra0Q6nOd2KbAouMs0g2G55CBKFPs3IInn4t5n0iq035YzK7HdnAmAqjoECvOGTV4i3m79DyY1PfU='} 2024-04-10T19:59:20.467 DEBUG:teuthology.nuke:shortname: smithi040 2024-04-10T19:59:20.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:59:20.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:23:59.312755', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPjJ4nhRr+25DGl/pRQUcmtZCaiEcVmIuqjFZf093bRN6+vT+mut3s5Sxb6QgNyn/gSBPPeP6Qyx+DWqRNcvS6U='} 2024-04-10T19:59:20.532 DEBUG:teuthology.nuke:shortname: smithi117 2024-04-10T19:59:20.533 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T19:59:20.582 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7651055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 19:23:59.309280', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIWlv5zdgEbJ5eclJjXBh/QTJjTB/pHlM+ouDzkvuMCNObNjzzCDOIhfhTchCxHQyoadh39PUCZXHXd1jkQkQlk='} 2024-04-10T19:59:20.603 INFO:teuthology.orchestra.console.smithi028:Power off 2024-04-10T19:59:20.603 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T19:59:20.652 INFO:teuthology.orchestra.console.smithi040:Power off 2024-04-10T19:59:20.652 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T19:59:20.676 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-04-10T19:59:20.676 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:20.703 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-04-10T19:59:20.703 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:20.716 INFO:teuthology.orchestra.console.smithi117:Power off 2024-04-10T19:59:20.716 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T19:59:20.731 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-10T19:59:20.755 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-10T19:59:20.767 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-04-10T19:59:20.767 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:20.819 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-04-10T19:59:24.732 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:24.756 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:24.819 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:24.843 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-10T19:59:24.865 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-10T19:59:24.928 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-04-10T19:59:28.847 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:28.866 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:28.929 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:28.958 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-04-10T19:59:28.975 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-04-10T19:59:29.037 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-04-10T19:59:29.059 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-04-10T19:59:29.076 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-04-10T19:59:33.041 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T19:59:33.151 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-04-10T19:59:33.252 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-04-10T19:59:33.358 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} duration: 1088.4839675426483 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=537570e869f84fdfad8514d55867f48f status: dead success: false 2024-04-10T19:59:33.358 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T19:59:33.471 INFO:teuthology.run:DEAD