2024-04-10T16:22:19.131 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T16:22:19.131 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745 --verbose -- /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745/orig.config.yaml 2024-04-10T16:22:19.178 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T16:22:19.269 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745 branch: wip-yuri6-testing-2024-04-02-1310 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} email: yweinste@redhat.com first_in_suite: false job_id: '7650745' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2024-04-02-1310 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum mon_bind_msgr2: false sha1: a5074d4516d566e9d8b6aec912f26afd099de101 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a5074d4516d566e9d8b6aec912f26afd099de101 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri6-testing-2024-04-02-1310 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 4372 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2024-04-02-1310 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a5074d4516d566e9d8b6aec912f26afd099de101 targets: smithi057.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMLg4Q5cfyrFGfSTTko1juS/HaftKqF8SgHnKusjmWrthMbRdOwQ4kKgoISiCcPuctth9zfoU4AiP8DH3pGQfYM= smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8El2dLMP7fgZBU+1QJgmwsgUWWGPnettXXdm0P12FzPOoDZkJaTD1J/B3eIt+QNl4uqSS9051iyoy6gopZuBo= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFr17ZxEh24kZ94VYkluWVZMyD+Zc8wcHFDtAYQ3xdHEjH8PwM8wDQWnJJHPTT3Q9Uo4FBBrAZjZnzDvaMsnZKU= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:17:51 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T16:22:19.269 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa; will attempt to use it 2024-04-10T16:22:19.270 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa/tasks 2024-04-10T16:22:19.270 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T16:22:19.271 INFO:teuthology.task.internal:Saving configuration 2024-04-10T16:22:19.287 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T16:22:19.294 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T16:22:19.318 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:16:28.543758', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMLg4Q5cfyrFGfSTTko1juS/HaftKqF8SgHnKusjmWrthMbRdOwQ4kKgoISiCcPuctth9zfoU4AiP8DH3pGQfYM='} 2024-04-10T16:22:19.340 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:16:28.542557', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8El2dLMP7fgZBU+1QJgmwsgUWWGPnettXXdm0P12FzPOoDZkJaTD1J/B3eIt+QNl4uqSS9051iyoy6gopZuBo='} 2024-04-10T16:22:19.359 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:16:28.540848', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFr17ZxEh24kZ94VYkluWVZMyD+Zc8wcHFDtAYQ3xdHEjH8PwM8wDQWnJJHPTT3Q9Uo4FBBrAZjZnzDvaMsnZKU='} 2024-04-10T16:22:19.359 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T16:22:19.365 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T16:22:19.365 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T16:22:19.365 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T16:22:19.365 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T16:22:19.467 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f633609cf70>, signals=[15]) 2024-04-10T16:22:19.468 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T16:22:19.474 INFO:teuthology.task.internal:Opening connections... 2024-04-10T16:22:19.474 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2024-04-10T16:22:19.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:22:19.554 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-04-10T16:22:19.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:22:19.627 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-04-10T16:22:19.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:22:19.700 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T16:22:19.707 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2024-04-10T16:22:19.730 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2024-04-10T16:22:19.731 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2024-04-10T16:22:19.793 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream" 2024-04-10T16:22:19.793 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="8" 2024-04-10T16:22:19.793 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos" 2024-04-10T16:22:19.793 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora" 2024-04-10T16:22:19.793 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="8" 2024-04-10T16:22:19.793 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el8" 2024-04-10T16:22:19.793 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T16:22:19.793 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2024-04-10T16:22:19.794 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T16:22:19.794 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/" 2024-04-10T16:22:19.794 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T16:22:19.794 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T16:22:19.794 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T16:22:19.794 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2024-04-10T16:22:20.020 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-04-10T16:22:20.042 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-04-10T16:22:20.042 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-04-10T16:22:20.102 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T16:22:20.103 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T16:22:20.103 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T16:22:20.103 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-04-10T16:22:20.124 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-04-10T16:22:20.147 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-04-10T16:22:20.147 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-04-10T16:22:20.208 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-04-10T16:22:20.208 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="8" 2024-04-10T16:22:20.208 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-04-10T16:22:20.208 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-04-10T16:22:20.209 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="8" 2024-04-10T16:22:20.209 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el8" 2024-04-10T16:22:20.209 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T16:22:20.209 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-04-10T16:22:20.209 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T16:22:20.209 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-04-10T16:22:20.209 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T16:22:20.209 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T16:22:20.209 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T16:22:20.209 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-04-10T16:22:20.233 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T16:22:20.242 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T16:22:20.248 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T16:22:20.249 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2024-04-10T16:22:20.251 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-04-10T16:22:20.253 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-04-10T16:22:20.273 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T16:22:20.279 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T16:22:20.280 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2024-04-10T16:22:20.314 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-04-10T16:22:20.316 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-04-10T16:22:20.354 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T16:22:20.431 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T16:22:20.446 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T16:22:20.446 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T16:22:20.446 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T16:22:20.447 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T16:22:20.447 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T16:22:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:22:20.467 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-04-10T16:22:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:22:20.468 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-04-10T16:22:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:22:20.469 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2024-04-10T16:22:20.527 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-489.el8.x86_64 2024-04-10T16:22:20.527 INFO:teuthology.task.kernel:Running kernel on smithi131: 4.18.0-489.el8.x86_64 2024-04-10T16:22:20.527 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum install -y kernel 2024-04-10T16:22:20.530 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-489.el8.x86_64 2024-04-10T16:22:20.530 INFO:teuthology.task.kernel:Running kernel on smithi107: 4.18.0-489.el8.x86_64 2024-04-10T16:22:20.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-04-10T16:22:20.531 INFO:teuthology.orchestra.run.smithi057.stdout:4.18.0-489.el8.x86_64 2024-04-10T16:22:20.531 INFO:teuthology.task.kernel:Running kernel on smithi057: 4.18.0-489.el8.x86_64 2024-04-10T16:22:20.532 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum install -y kernel 2024-04-10T16:22:26.404 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - AppStream 8.2 MB/s | 28 MB 00:03 2024-04-10T16:22:28.093 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - AppStream 5.7 MB/s | 28 MB 00:04 2024-04-10T16:24:46.256 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - AppStream 200 kB/s | 28 MB 02:23 2024-04-10T16:24:46.503 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - BaseOS 78 kB/s | 10 MB 02:14 2024-04-10T16:24:47.150 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - BaseOS 79 kB/s | 10 MB 02:13 2024-04-10T16:24:49.842 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras 18 kB/s | 18 kB 00:00 2024-04-10T16:24:55.206 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - BaseOS 2.6 MB/s | 10 MB 00:03 2024-04-10T16:24:57.757 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras 19 kB/s | 18 kB 00:00 2024-04-10T16:25:42.610 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras common packages 176 B/s | 7.7 kB 00:44 2024-04-10T16:27:00.910 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras 137 B/s | 18 kB 02:12 2024-04-10T16:27:11.048 INFO:teuthology.orchestra.run.smithi131.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T16:27:11.093 INFO:teuthology.orchestra.run.smithi131.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T16:27:11.093 INFO:teuthology.orchestra.run.smithi131.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T16:27:11.093 INFO:teuthology.orchestra.run.smithi131.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T16:27:15.375 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 3.8 MB/s | 16 MB 00:04 2024-04-10T16:27:19.116 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 415 kB/s | 24 kB 00:00 2024-04-10T16:27:19.258 INFO:teuthology.orchestra.run.smithi131.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T16:27:22.080 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T16:27:22.080 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T16:27:22.201 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-04-10T16:27:22.202 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-04-10T16:27:22.202 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-04-10T16:27:22.202 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-04-10T16:27:22.202 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-04-10T16:27:22.202 INFO:teuthology.orchestra.run.smithi131.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T16:27:22.202 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-04-10T16:27:22.202 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T16:27:22.203 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T16:27:22.203 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T16:27:22.203 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-04-10T16:27:22.203 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-04-10T16:27:22.203 INFO:teuthology.orchestra.run.smithi131.stdout:Install 3 Packages 2024-04-10T16:27:22.203 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T16:27:22.203 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 90 M 2024-04-10T16:27:22.204 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 96 M 2024-04-10T16:27:22.204 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-04-10T16:27:24.510 INFO:teuthology.orchestra.run.smithi131.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 5.1 MB/s | 10 MB 00:02 2024-04-10T16:27:26.293 INFO:teuthology.orchestra.run.smithi131.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 12 MB/s | 44 MB 00:03 2024-04-10T16:27:26.772 INFO:teuthology.orchestra.run.smithi131.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 8.5 MB/s | 36 MB 00:04 2024-04-10T16:27:26.772 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-04-10T16:27:26.772 INFO:teuthology.orchestra.run.smithi131.stdout:Total 20 MB/s | 90 MB 00:04 2024-04-10T16:27:27.459 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-04-10T16:27:27.910 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-04-10T16:27:27.910 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-04-10T16:27:29.151 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-04-10T16:27:29.177 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-04-10T16:27:32.106 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-04-10T16:27:32.663 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T16:27:36.305 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T16:27:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T16:27:40.129 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T16:27:40.462 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:27:43.061 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras common packages 187 B/s | 7.7 kB 00:42 2024-04-10T16:27:43.095 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras common packages 46 B/s | 7.7 kB 02:53 2024-04-10T16:28:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:28:02.812 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:28:03.193 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:28:03.193 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T16:28:03.193 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T16:28:04.175 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:28:04.176 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T16:28:04.176 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-04-10T16:28:04.176 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T16:28:04.176 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T16:28:04.176 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T16:28:04.176 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-04-10T16:28:04.334 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T16:28:04.551 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T16:28:04.552 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T16:28:04.552 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T16:28:04.552 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T16:28:04.552 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T16:28:04.552 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T16:28:04.552 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T16:28:04.552 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T16:28:04.763 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T16:28:04.763 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T16:28:04.764 DEBUG:teuthology.orchestra.run.smithi131:> sudo rpm -qi grub2-tools 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Name : grub2-tools 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Epoch : 1 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Version : 2.02 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Release : 129.el8 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: x86_64 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Group : System Environment/Base 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Size : 9265586 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:License : GPLv3+ 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T16:28:04.798 INFO:teuthology.orchestra.run.smithi131.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:Relocations : (not relocatable) 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:Packager : CentOS BuildSystem 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:Vendor : CentOS 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:Summary : Support tools for GRUB. 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:Description : 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout:hardware devices. 2024-04-10T16:28:04.799 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-10T16:28:04.800 INFO:teuthology.orchestra.run.smithi131.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T16:28:04.800 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T16:28:04.800 INFO:teuthology.task.kernel:Updating grub on smithi131 to boot 4.18.0-552.el8.x86_64 2024-04-10T16:28:04.801 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T16:28:06.287 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2024-04-10T16:28:08.522 INFO:teuthology.orchestra.run.smithi131.stderr:done 2024-04-10T16:28:08.524 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2024-04-10T16:28:08.556 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.fDmziVegYC 2024-04-10T16:28:08.556 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fDmziVegYC 2024-04-10T16:28:08.630 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.fDmziVegYC 2024-04-10T16:28:08.778 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.fDmziVegYC is 6KB 2024-04-10T16:28:08.827 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.fDmziVegYC 2024-04-10T16:28:08.844 DEBUG:teuthology.orchestra.run.smithi131:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T16:28:08.912 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T16:28:08.912 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T16:28:08.912 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T16:28:08.912 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T16:28:08.914 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T16:28:09.119 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2024-04-10T16:28:39.125 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T16:28:39.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T16:28:39.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:29:10.507 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:27 2024-04-10T16:29:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T16:29:10.508 INFO:teuthology.orchestra.run.smithi107.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T16:29:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T16:29:10.555 INFO:teuthology.orchestra.run.smithi057.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:27 2024-04-10T16:29:10.556 INFO:teuthology.orchestra.run.smithi057.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T16:29:10.556 INFO:teuthology.orchestra.run.smithi057.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T16:29:10.556 INFO:teuthology.orchestra.run.smithi057.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T16:29:14.048 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 16 MB 00:03 2024-04-10T16:29:17.786 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 411 kB/s | 24 kB 00:00 2024-04-10T16:29:17.927 INFO:teuthology.orchestra.run.smithi107.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T16:29:20.724 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T16:29:20.724 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T16:29:20.834 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-10T16:29:20.834 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-10T16:29:20.835 INFO:teuthology.orchestra.run.smithi107.stdout:Install 3 Packages 2024-04-10T16:29:20.836 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T16:29:20.836 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 90 M 2024-04-10T16:29:20.836 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 96 M 2024-04-10T16:29:20.836 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-10T16:29:39.129 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T16:29:48.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T16:29:48.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:29:49.460 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-04-10T16:29:49.943 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-10T16:29:49.943 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T16:29:49.944 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-04-10T16:29:49.959 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-552.el8.x86_64 2024-04-10T16:29:49.960 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T16:29:49.960 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T16:29:49.960 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T16:29:50.961 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T16:29:50.961 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T16:29:51.030 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2024-04-10T16:29:51.043 DEBUG:teuthology.parallel:result is None 2024-04-10T16:31:32.412 INFO:teuthology.orchestra.run.smithi107.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 5.2 MB/s | 10 MB 00:02 2024-04-10T16:31:34.067 INFO:teuthology.orchestra.run.smithi107.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 10 MB/s | 36 MB 00:03 2024-04-10T16:31:34.478 INFO:teuthology.orchestra.run.smithi107.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 11 MB/s | 44 MB 00:04 2024-04-10T16:31:34.479 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-10T16:31:34.479 INFO:teuthology.orchestra.run.smithi107.stdout:Total 693 kB/s | 90 MB 02:13 2024-04-10T16:31:35.160 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-10T16:31:35.581 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-10T16:31:35.581 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-10T16:31:36.846 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-10T16:31:36.873 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-10T16:31:39.669 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-10T16:31:40.258 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T16:31:43.839 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T16:31:44.206 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T16:31:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T16:31:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:32:09.740 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:32:09.763 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:32:10.096 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:32:10.096 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T16:32:10.096 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T16:32:10.613 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:32:10.613 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T16:32:10.613 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-10T16:32:10.613 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T16:32:10.613 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T16:32:10.614 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T16:32:10.614 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-10T16:32:10.750 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T16:32:11.011 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T16:32:11.011 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T16:32:11.011 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T16:32:11.012 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T16:32:11.012 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T16:32:11.012 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T16:32:11.012 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T16:32:11.012 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T16:32:11.222 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T16:32:11.222 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T16:32:11.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-04-10T16:32:11.256 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-04-10T16:32:11.256 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-04-10T16:32:11.256 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.02 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 129.el8 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Group : System Environment/Base 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 9265586 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Relocations : (not relocatable) 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : CentOS BuildSystem 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-04-10T16:32:11.257 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T16:32:11.258 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-04-10T16:32:11.258 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-04-10T16:32:11.258 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T16:32:11.258 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T16:32:11.258 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T16:32:11.258 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T16:32:11.258 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-04-10T16:32:11.258 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-10T16:32:11.258 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T16:32:11.259 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T16:32:11.259 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 4.18.0-552.el8.x86_64 2024-04-10T16:32:11.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T16:32:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-04-10T16:32:15.092 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-04-10T16:32:15.094 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-04-10T16:32:15.116 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.DUsWpx1OSh 2024-04-10T16:32:15.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DUsWpx1OSh 2024-04-10T16:32:15.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.DUsWpx1OSh 2024-04-10T16:32:15.301 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.DUsWpx1OSh is 6KB 2024-04-10T16:32:15.349 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.DUsWpx1OSh 2024-04-10T16:32:15.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T16:32:15.440 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T16:32:15.440 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T16:32:15.440 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T16:32:15.440 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T16:32:15.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T16:32:15.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-04-10T16:32:45.608 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T16:32:45.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-10T16:32:45.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:33:20.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-10T16:33:29.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-10T16:33:29.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:33:47.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-10T16:33:59.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-10T16:33:59.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:33:59.855 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-04-10T16:34:00.308 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-10T16:34:00.308 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T16:34:00.308 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-04-10T16:34:00.324 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-552.el8.x86_64 2024-04-10T16:34:00.325 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T16:34:00.325 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T16:34:00.325 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T16:34:01.325 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T16:34:01.326 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T16:34:01.404 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-04-10T16:34:01.419 DEBUG:teuthology.parallel:result is None 2024-04-10T16:37:55.984 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 32 kB/s | 16 MB 08:45 2024-04-10T16:37:59.948 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 258 kB/s | 24 kB 00:00 2024-04-10T16:38:00.103 INFO:teuthology.orchestra.run.smithi057.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T16:38:03.062 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T16:38:03.062 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T16:38:03.174 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-04-10T16:38:03.175 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-04-10T16:38:03.175 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2024-04-10T16:38:03.175 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-04-10T16:38:03.175 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-04-10T16:38:03.175 INFO:teuthology.orchestra.run.smithi057.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T16:38:03.175 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-04-10T16:38:03.175 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T16:38:03.175 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T16:38:03.176 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-10T16:38:03.176 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-04-10T16:38:03.176 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-04-10T16:38:03.176 INFO:teuthology.orchestra.run.smithi057.stdout:Install 3 Packages 2024-04-10T16:38:03.176 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-10T16:38:03.176 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 90 M 2024-04-10T16:38:03.176 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 96 M 2024-04-10T16:38:03.176 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-04-10T16:38:11.308 INFO:teuthology.orchestra.run.smithi057.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 1.7 MB/s | 10 MB 00:06 2024-04-10T16:38:14.307 INFO:teuthology.orchestra.run.smithi057.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 4.0 MB/s | 36 MB 00:09 2024-04-10T16:38:14.891 INFO:teuthology.orchestra.run.smithi057.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 4.5 MB/s | 44 MB 00:09 2024-04-10T16:38:14.891 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-04-10T16:38:14.891 INFO:teuthology.orchestra.run.smithi057.stdout:Total 7.7 MB/s | 90 MB 00:11 2024-04-10T16:38:15.603 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-04-10T16:38:16.056 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-04-10T16:38:16.056 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-04-10T16:38:17.370 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-04-10T16:38:17.419 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-04-10T16:38:20.428 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-04-10T16:38:20.989 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T16:38:24.744 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T16:38:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T16:38:28.855 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T16:38:29.191 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:38:52.728 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:38:52.754 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:38:53.154 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:38:53.154 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T16:38:53.154 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T16:38:53.703 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T16:38:53.703 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-10T16:38:53.703 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-04-10T16:38:53.703 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T16:38:53.703 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T16:38:53.703 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-10T16:38:53.703 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-04-10T16:38:53.860 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T16:38:54.092 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T16:38:54.092 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T16:38:54.092 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T16:38:54.093 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T16:38:54.093 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T16:38:54.093 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T16:38:54.093 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T16:38:54.093 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T16:38:54.314 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T16:38:54.315 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T16:38:54.315 DEBUG:teuthology.orchestra.run.smithi057:> sudo rpm -qi grub2-tools 2024-04-10T16:38:54.349 INFO:teuthology.orchestra.run.smithi057.stdout:Name : grub2-tools 2024-04-10T16:38:54.349 INFO:teuthology.orchestra.run.smithi057.stdout:Epoch : 1 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Version : 2.02 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Release : 129.el8 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: x86_64 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Group : System Environment/Base 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Size : 9265586 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:License : GPLv3+ 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Relocations : (not relocatable) 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Packager : CentOS BuildSystem 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:Vendor : CentOS 2024-04-10T16:38:54.350 INFO:teuthology.orchestra.run.smithi057.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T16:38:54.351 INFO:teuthology.orchestra.run.smithi057.stdout:Summary : Support tools for GRUB. 2024-04-10T16:38:54.351 INFO:teuthology.orchestra.run.smithi057.stdout:Description : 2024-04-10T16:38:54.351 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-10T16:38:54.351 INFO:teuthology.orchestra.run.smithi057.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T16:38:54.351 INFO:teuthology.orchestra.run.smithi057.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T16:38:54.351 INFO:teuthology.orchestra.run.smithi057.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T16:38:54.351 INFO:teuthology.orchestra.run.smithi057.stdout:hardware devices. 2024-04-10T16:38:54.351 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-10T16:38:54.351 INFO:teuthology.orchestra.run.smithi057.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T16:38:54.352 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T16:38:54.352 INFO:teuthology.task.kernel:Updating grub on smithi057 to boot 4.18.0-552.el8.x86_64 2024-04-10T16:38:54.352 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T16:38:55.829 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2024-04-10T16:38:58.065 INFO:teuthology.orchestra.run.smithi057.stderr:done 2024-04-10T16:38:58.067 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2024-04-10T16:38:58.098 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.YPYljJTora 2024-04-10T16:38:58.098 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YPYljJTora 2024-04-10T16:38:58.173 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.YPYljJTora 2024-04-10T16:38:58.318 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.YPYljJTora is 6KB 2024-04-10T16:38:58.367 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.YPYljJTora 2024-04-10T16:38:58.384 DEBUG:teuthology.orchestra.run.smithi057:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T16:38:58.459 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T16:38:58.459 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T16:38:58.459 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T16:38:58.459 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T16:38:58.461 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T16:38:58.756 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2024-04-10T16:39:28.759 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T16:39:28.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-04-10T16:39:28.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:39:47.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2024-04-10T16:39:56.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-04-10T16:39:56.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:39:59.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2024-04-10T16:40:11.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-04-10T16:40:11.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:41:11.428 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T16:41:26.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-04-10T16:41:26.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T16:41:26.864 DEBUG:teuthology.orchestra.run.smithi057:> true 2024-04-10T16:41:27.199 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-04-10T16:41:27.199 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T16:41:27.200 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2024-04-10T16:41:27.220 INFO:teuthology.orchestra.run.smithi057.stdout:4.18.0-552.el8.x86_64 2024-04-10T16:41:27.220 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T16:41:27.221 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T16:41:27.221 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T16:41:28.222 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T16:41:28.222 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T16:41:28.336 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2024-04-10T16:41:28.352 DEBUG:teuthology.parallel:result is None 2024-04-10T16:41:28.352 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T16:41:28.362 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T16:41:28.362 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T16:41:28.365 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T16:41:28.368 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T16:41:28.391 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T16:41:28.400 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T16:41:28.428 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T16:41:28.428 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T16:41:28.430 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T16:41:28.433 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T16:41:28.479 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T16:41:28.490 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T16:41:28.491 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T16:41:28.502 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T16:41:28.516 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T16:41:28.550 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T16:41:28.557 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T16:41:28.558 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T16:41:28.562 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T16:41:28.568 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T16:41:28.570 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T16:41:28.571 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T16:41:28.581 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T16:41:28.582 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T16:41:28.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T16:41:28.586 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T16:41:28.641 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T16:41:28.654 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T16:41:28.654 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T16:41:28.656 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T16:41:28.659 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T16:41:28.701 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T16:41:28.829 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T16:41:28.866 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T16:41:29.026 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T16:41:29.058 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-10T16:41:29.058 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T16:41:29.129 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T16:41:29.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T16:41:29.294 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T16:41:29.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T16:41:29.497 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-10T16:41:29.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T16:41:29.569 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T16:41:29.722 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T16:41:29.771 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T16:41:29.927 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T16:41:29.958 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-10T16:41:29.958 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T16:41:30.028 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2024-04-10T16:41:30.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-04-10T16:41:30.033 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-04-10T16:41:30.097 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T16:41:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T16:41:30.147 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T16:41:30.565 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T16:41:30.572 INFO:teuthology.task.internal:Starting timer... 2024-04-10T16:41:30.572 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T16:41:30.687 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T16:41:30.737 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T16:41:30.737 DEBUG:teuthology.orchestra.run.smithi057:> sudo service auditd rotate 2024-04-10T16:41:30.825 INFO:teuthology.orchestra.run.smithi057.stdout:Rotating logs: 2024-04-10T16:41:30.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-04-10T16:41:30.896 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-04-10T16:41:30.898 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2024-04-10T16:41:30.973 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2024-04-10T16:41:30.975 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T16:41:30.975 DEBUG:teuthology.orchestra.run.smithi057:> /usr/sbin/getenforce 2024-04-10T16:41:31.042 INFO:teuthology.orchestra.run.smithi057.stdout:Permissive 2024-04-10T16:41:31.042 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-04-10T16:41:31.076 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-04-10T16:41:31.077 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2024-04-10T16:41:31.099 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2024-04-10T16:41:31.099 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi057.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi131.front.sepia.ceph.com': 'permissive'} 2024-04-10T16:41:31.100 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T16:41:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:41:31.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T16:41:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:41:31.158 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T16:41:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:41:31.185 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T16:41:31.186 DEBUG:teuthology.orchestra.run.smithi057:> sudo /usr/sbin/setenforce permissive 2024-04-10T16:41:31.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-04-10T16:41:31.250 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2024-04-10T16:41:31.278 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T16:41:31.288 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T16:41:31.343 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T16:41:31.782 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T16:41:31.847 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T16:41:31.849 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com,smithi107.front.sepia.ceph.com,smithi131.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T16:59:42.765 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745/ansible_failures.yaml 2024-04-10T16:59:42.768 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T16:59:42.985 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ef1b0e06ecd84b0d991a6c5c4d554f26 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T16:59:42.987 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T16:59:42.996 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T16:59:42.996 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T16:59:43.009 DEBUG:teuthology.orchestra.run.smithi057:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T16:59:43.082 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T16:59:43.146 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T16:59:43.210 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T16:59:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:59:43.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T16:59:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:59:43.277 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T16:59:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:59:43.311 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T16:59:43.319 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T16:59:43.328 INFO:teuthology.task.internal:Duration was 1092.755612 seconds 2024-04-10T16:59:43.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T16:59:43.337 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T16:59:43.337 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T16:59:43.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T16:59:43.341 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T16:59:43.389 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T16:59:43.391 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T16:59:43.402 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T16:59:43.819 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T16:59:43.819 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2024-04-10T16:59:43.819 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T16:59:43.846 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-04-10T16:59:43.847 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T16:59:43.875 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-04-10T16:59:43.875 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T16:59:43.902 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T16:59:43.902 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T16:59:43.904 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T16:59:43.917 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T16:59:43.984 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T16:59:43.985 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T16:59:43.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T16:59:44.000 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T16:59:44.089 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T16:59:44.097 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T16:59:44.097 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T16:59:44.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T16:59:44.153 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T16:59:44.181 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T16:59:44.189 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T16:59:44.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T16:59:44.195 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T16:59:44.216 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2024-04-10T16:59:44.218 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-04-10T16:59:44.248 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-04-10T16:59:44.269 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T16:59:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:59:44.305 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T16:59:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:59:44.321 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T16:59:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:59:44.339 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T16:59:44.347 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T16:59:44.349 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745/remote/smithi057 2024-04-10T16:59:44.350 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T16:59:44.446 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745/remote/smithi107 2024-04-10T16:59:44.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T16:59:44.516 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745/remote/smithi131 2024-04-10T16:59:44.517 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T16:59:44.576 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T16:59:44.576 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T16:59:44.578 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T16:59:44.581 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T16:59:44.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T16:59:44.608 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T16:59:44.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T16:59:44.616 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T16:59:44.616 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T16:59:44.636 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T16:59:44.638 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T16:59:44.652 INFO:teuthology.orchestra.run.smithi057.stdout: 525354 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 16:59 /home/ubuntu/cephtest 2024-04-10T16:59:44.653 INFO:teuthology.orchestra.run.smithi107.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 16:59 /home/ubuntu/cephtest 2024-04-10T16:59:44.656 INFO:teuthology.orchestra.run.smithi131.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 16:59 /home/ubuntu/cephtest 2024-04-10T16:59:44.657 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T16:59:44.665 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T16:59:44.689 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T16:59:44.715 DEBUG:teuthology.nuke:shortname: smithi057 2024-04-10T16:59:44.715 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T16:59:44.736 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:16:28.543758', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMLg4Q5cfyrFGfSTTko1juS/HaftKqF8SgHnKusjmWrthMbRdOwQ4kKgoISiCcPuctth9zfoU4AiP8DH3pGQfYM='} 2024-04-10T16:59:44.740 DEBUG:teuthology.nuke:shortname: smithi107 2024-04-10T16:59:44.740 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T16:59:44.766 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:16:28.542557', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8El2dLMP7fgZBU+1QJgmwsgUWWGPnettXXdm0P12FzPOoDZkJaTD1J/B3eIt+QNl4uqSS9051iyoy6gopZuBo='} 2024-04-10T16:59:44.767 DEBUG:teuthology.nuke:shortname: smithi131 2024-04-10T16:59:44.767 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T16:59:44.793 INFO:teuthology.orchestra.console.smithi057:Power off 2024-04-10T16:59:44.793 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T16:59:44.805 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 16:16:28.540848', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFr17ZxEh24kZ94VYkluWVZMyD+Zc8wcHFDtAYQ3xdHEjH8PwM8wDQWnJJHPTT3Q9Uo4FBBrAZjZnzDvaMsnZKU='} 2024-04-10T16:59:44.818 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2024-04-10T16:59:44.818 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:44.833 INFO:teuthology.orchestra.console.smithi107:Power off 2024-04-10T16:59:44.833 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T16:59:44.845 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-04-10T16:59:44.856 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-04-10T16:59:44.856 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:44.875 INFO:teuthology.orchestra.console.smithi131:Power off 2024-04-10T16:59:44.875 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T16:59:44.886 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-10T16:59:44.900 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-04-10T16:59:44.900 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:44.926 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-04-10T16:59:48.847 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:48.887 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:48.927 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:48.956 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-04-10T16:59:48.996 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-10T16:59:49.037 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-04-10T16:59:52.960 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:52.997 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:53.038 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:53.070 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2024-04-10T16:59:53.107 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-10T16:59:53.147 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-04-10T16:59:53.172 INFO:teuthology.orchestra.console.smithi057:Power off completed 2024-04-10T16:59:53.249 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-04-10T16:59:57.112 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:59:57.221 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-04-10T16:59:57.322 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-04-10T16:59:57.427 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} duration: 1092.7556116580963 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ef1b0e06ecd84b0d991a6c5c4d554f26 status: dead success: false 2024-04-10T16:59:57.428 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T16:59:57.515 INFO:teuthology.run:DEAD