2024-02-18T00:22:27.221 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-18T00:22:27.221 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564972 --owner scheduled_yuriw@teuthology --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} --verbose --name yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564972/orig.config.yaml 2024-02-18T00:22:27.262 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-18T00:22:27.359 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564972 branch: pacific-release description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: yweinste@redhat.com first_in_suite: false job_id: '7564972' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_async_recovery_min_cost: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - \(MON_DOWN\) - osds down - mons down - flag\(s\) set - out of quorum - PG_ - Reduced data availability - stuck undersized sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: pacific-release sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 264 sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: pacific-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLU4NWa2JJ9RrHp2d5NlizlOoZ3xs1JfjW88RxRBqdYPCMd4mZA/5Hlf0vBcqJilulqmRvkVBaSnFglKh0RawBI= smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEn6z+awfxqplhzV//lO7go10/J54r1co/CrvVwHkXYmxT4wqfDyrskZBIs8rc2FamGpS7erTo/qGB+3umrhI1U= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf/R4z+6X5OwmVZAQTBGfm+VQadRCLX1Pq2P+G7sZDjfqZka+6QQ0tsPQ74jW7eAMAGz2A1U1X+bLfYUWobOx0= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBON54F23dLfDwlQHxNl9IVQ5sgUEuhnSFmGcT57OzIElOaVQjyxJ4dwcqvwqxZbS7KuMyG8wz6GRcDOT8DTbLdU= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-17_15:55:51 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-18T00:22:27.359 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa; will attempt to use it 2024-02-18T00:22:27.360 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks 2024-02-18T00:22:27.360 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-18T00:22:27.361 INFO:teuthology.task.internal:Checking packages... 2024-02-18T00:22:27.388 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '81bd20d634209c7cb82c18be12b4b5a05643ebf1' 2024-02-18T00:22:27.388 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-18T00:22:27.388 INFO:teuthology.packaging:ref: None 2024-02-18T00:22:27.388 INFO:teuthology.packaging:tag: None 2024-02-18T00:22:27.388 INFO:teuthology.packaging:branch: pacific-release 2024-02-18T00:22:27.388 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-18T00:22:27.388 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-18T00:22:27.527 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-548.g81bd20d6 2024-02-18T00:22:27.529 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-18T00:22:27.534 INFO:teuthology.task.internal:no buildpackages task found 2024-02-18T00:22:27.534 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-18T00:22:27.563 INFO:teuthology.task.internal:Saving configuration 2024-02-18T00:22:27.579 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-18T00:22:27.613 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-18T00:22:27.638 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-18 00:17:12.931580', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLU4NWa2JJ9RrHp2d5NlizlOoZ3xs1JfjW88RxRBqdYPCMd4mZA/5Hlf0vBcqJilulqmRvkVBaSnFglKh0RawBI='} 2024-02-18T00:22:27.658 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564973', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-18 00:17:13.390151', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEn6z+awfxqplhzV//lO7go10/J54r1co/CrvVwHkXYmxT4wqfDyrskZBIs8rc2FamGpS7erTo/qGB+3umrhI1U='} 2024-02-18T00:22:27.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-18 00:17:12.934478', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf/R4z+6X5OwmVZAQTBGfm+VQadRCLX1Pq2P+G7sZDjfqZka+6QQ0tsPQ74jW7eAMAGz2A1U1X+bLfYUWobOx0='} 2024-02-18T00:22:27.702 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564973', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-18 00:17:13.388302', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBON54F23dLfDwlQHxNl9IVQ5sgUEuhnSFmGcT57OzIElOaVQjyxJ4dwcqvwqxZbS7KuMyG8wz6GRcDOT8DTbLdU='} 2024-02-18T00:22:27.702 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-18T00:22:27.708 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-02-18T00:22:27.708 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-02-18T00:22:27.708 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-02-18T00:22:27.708 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-02-18T00:22:27.708 INFO:teuthology.run_tasks:Running task console_log... 2024-02-18T00:22:27.877 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4229e840d0>, signals=[15]) 2024-02-18T00:22:27.878 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-18T00:22:27.883 INFO:teuthology.task.internal:Opening connections... 2024-02-18T00:22:27.883 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-02-18T00:22:27.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-18T00:22:27.964 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-02-18T00:22:27.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-18T00:22:28.040 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-02-18T00:22:28.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-18T00:22:28.117 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-02-18T00:22:28.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-18T00:22:28.195 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-18T00:22:28.202 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-02-18T00:22:28.225 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-02-18T00:22:28.225 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-02-18T00:22:28.286 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-02-18T00:22:28.286 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="8" 2024-02-18T00:22:28.286 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-02-18T00:22:28.286 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-02-18T00:22:28.286 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="8" 2024-02-18T00:22:28.286 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el8" 2024-02-18T00:22:28.286 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-18T00:22:28.287 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-02-18T00:22:28.287 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-18T00:22:28.287 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-02-18T00:22:28.287 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-18T00:22:28.287 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-18T00:22:28.287 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-18T00:22:28.288 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-02-18T00:22:28.316 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-02-18T00:22:28.337 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-02-18T00:22:28.337 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-02-18T00:22:28.400 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-02-18T00:22:28.400 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="8" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="8" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el8" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-18T00:22:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-18T00:22:28.401 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-02-18T00:22:28.424 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-02-18T00:22:28.447 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-02-18T00:22:28.448 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="8" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="8" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el8" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-02-18T00:22:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-18T00:22:28.511 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-18T00:22:28.511 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-18T00:22:28.511 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-02-18T00:22:28.532 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-02-18T00:22:28.554 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-02-18T00:22:28.555 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-02-18T00:22:28.615 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-02-18T00:22:28.615 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8" 2024-02-18T00:22:28.615 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-02-18T00:22:28.615 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-02-18T00:22:28.616 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8" 2024-02-18T00:22:28.616 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8" 2024-02-18T00:22:28.616 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-18T00:22:28.616 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-02-18T00:22:28.616 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-18T00:22:28.616 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-02-18T00:22:28.616 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-18T00:22:28.616 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-18T00:22:28.616 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-18T00:22:28.616 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-02-18T00:22:28.641 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-18T00:22:28.650 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-18T00:22:28.689 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-18T00:22:28.689 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-02-18T00:22:28.692 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-02-18T00:22:28.695 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-02-18T00:22:28.699 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-02-18T00:22:28.720 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-18T00:22:28.739 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-18T00:22:28.739 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-02-18T00:22:28.754 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-02-18T00:22:28.756 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-02-18T00:22:28.759 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-02-18T00:22:28.788 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-18T00:22:28.885 INFO:teuthology.run_tasks:Running task kernel... 2024-02-18T00:22:28.899 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-18T00:22:28.899 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-18T00:22:28.900 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-18T00:22:28.900 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-18T00:22:28.901 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-18T00:22:28.901 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-18T00:22:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-18T00:22:28.921 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-02-18T00:22:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-18T00:22:28.923 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-02-18T00:22:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-18T00:22:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-18T00:22:28.924 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-02-18T00:22:28.925 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-02-18T00:22:28.981 INFO:teuthology.orchestra.run.smithi193.stdout:4.18.0-489.el8.x86_64 2024-02-18T00:22:28.981 INFO:teuthology.task.kernel:Running kernel on smithi193: 4.18.0-489.el8.x86_64 2024-02-18T00:22:28.981 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum install -y kernel 2024-02-18T00:22:28.983 INFO:teuthology.orchestra.run.smithi151.stdout:4.18.0-489.el8.x86_64 2024-02-18T00:22:28.983 INFO:teuthology.task.kernel:Running kernel on smithi151: 4.18.0-489.el8.x86_64 2024-02-18T00:22:28.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-02-18T00:22:28.985 INFO:teuthology.orchestra.run.smithi184.stdout:4.18.0-489.el8.x86_64 2024-02-18T00:22:28.985 INFO:teuthology.task.kernel:Running kernel on smithi184: 4.18.0-489.el8.x86_64 2024-02-18T00:22:28.985 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2024-02-18T00:22:28.987 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-489.el8.x86_64 2024-02-18T00:22:28.987 INFO:teuthology.task.kernel:Running kernel on smithi038: 4.18.0-489.el8.x86_64 2024-02-18T00:22:28.987 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-02-18T00:22:29.227 INFO:teuthology.orchestra.run.smithi193.stdout:Waiting for process with pid 6842 to finish. 2024-02-18T00:22:29.239 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for process with pid 6796 to finish. 2024-02-18T00:22:32.646 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - AppStream 24 MB/s | 27 MB 00:01 2024-02-18T00:22:32.977 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - AppStream 21 MB/s | 27 MB 00:01 2024-02-18T00:22:39.346 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - BaseOS 6.5 MB/s | 10 MB 00:01 2024-02-18T00:22:39.652 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - BaseOS 5.5 MB/s | 10 MB 00:01 2024-02-18T00:22:41.305 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras 66 kB/s | 18 kB 00:00 2024-02-18T00:22:41.561 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - Extras 56 kB/s | 18 kB 00:00 2024-02-18T00:22:41.881 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2024-02-18T00:22:41.978 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2024-02-18T00:22:42.224 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-02-18T00:22:42.305 INFO:teuthology.orchestra.run.smithi184.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-02-18T00:22:43.380 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2024-02-18T00:22:43.919 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2024-02-18T00:22:45.760 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:01 ago on Sun 18 Feb 2024 12:22:44 AM UTC. 2024-02-18T00:22:47.214 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 332 kB/s | 24 kB 00:00 2024-02-18T00:22:47.933 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 443 kB/s | 24 kB 00:00 2024-02-18T00:22:48.039 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-18T00:22:48.039 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-18T00:22:48.082 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-18T00:22:48.083 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-18T00:22:48.083 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-02-18T00:22:48.083 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-18T00:22:48.083 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-18T00:22:48.083 INFO:teuthology.orchestra.run.smithi193.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-18T00:22:48.083 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-18T00:22:48.083 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-18T00:22:48.083 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-18T00:22:48.083 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-18T00:22:48.084 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-18T00:22:48.084 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-18T00:22:48.084 INFO:teuthology.orchestra.run.smithi193.stdout:Install 3 Packages 2024-02-18T00:22:48.084 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-18T00:22:48.194 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 90 M 2024-02-18T00:22:48.194 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 97 M 2024-02-18T00:22:48.194 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-18T00:22:48.195 INFO:teuthology.orchestra.run.smithi193.stdout:Waiting for process with pid 6842 to finish. 2024-02-18T00:22:50.144 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-18T00:22:50.144 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-18T00:22:50.257 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-18T00:22:50.258 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-18T00:22:50.259 INFO:teuthology.orchestra.run.smithi184.stdout:Install 3 Packages 2024-02-18T00:22:50.259 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-18T00:22:50.259 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 90 M 2024-02-18T00:22:50.259 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 97 M 2024-02-18T00:22:50.259 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-18T00:22:51.047 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-18T00:22:51.047 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-18T00:22:51.171 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-02-18T00:22:51.172 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-18T00:22:51.172 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-02-18T00:22:51.172 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-18T00:22:51.172 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-02-18T00:22:51.173 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-18T00:22:51.173 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-02-18T00:22:51.173 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-18T00:22:51.173 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-18T00:22:51.173 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-18T00:22:51.173 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-02-18T00:22:51.174 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-02-18T00:22:51.174 INFO:teuthology.orchestra.run.smithi038.stdout:Install 3 Packages 2024-02-18T00:22:51.174 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-18T00:22:51.175 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 90 M 2024-02-18T00:22:51.175 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 97 M 2024-02-18T00:22:51.175 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-02-18T00:22:51.199 INFO:teuthology.orchestra.run.smithi193.stdout:[SKIPPED] kernel-4.18.0-539.el8.x86_64.rpm: Already downloaded 2024-02-18T00:22:51.199 INFO:teuthology.orchestra.run.smithi193.stdout:[SKIPPED] kernel-core-4.18.0-539.el8.x86_64.rpm: Already downloaded 2024-02-18T00:22:51.199 INFO:teuthology.orchestra.run.smithi193.stdout:[SKIPPED] kernel-modules-4.18.0-539.el8.x86_64.rpm: Already downloaded 2024-02-18T00:22:51.765 INFO:teuthology.orchestra.run.smithi184.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 7.0 MB/s | 10 MB 00:01 2024-02-18T00:22:51.898 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-18T00:22:51.899 INFO:teuthology.orchestra.run.smithi193.stdout:Waiting for process with pid 6842 to finish. 2024-02-18T00:22:52.893 INFO:teuthology.orchestra.run.smithi038.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 6.1 MB/s | 10 MB 00:01 2024-02-18T00:22:53.332 INFO:teuthology.orchestra.run.smithi184.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 12 MB/s | 36 MB 00:03 2024-02-18T00:22:53.826 INFO:teuthology.orchestra.run.smithi038.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 14 MB/s | 36 MB 00:02 2024-02-18T00:22:54.476 INFO:teuthology.orchestra.run.smithi038.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 13 MB/s | 43 MB 00:03 2024-02-18T00:22:54.477 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-02-18T00:22:54.477 INFO:teuthology.orchestra.run.smithi038.stdout:Total 27 MB/s | 90 MB 00:03 2024-02-18T00:22:54.824 INFO:teuthology.orchestra.run.smithi184.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 9.5 MB/s | 43 MB 00:04 2024-02-18T00:22:54.824 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-18T00:22:54.824 INFO:teuthology.orchestra.run.smithi184.stdout:Total 20 MB/s | 90 MB 00:04 2024-02-18T00:22:55.195 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-02-18T00:22:55.502 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-18T00:22:55.635 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-02-18T00:22:55.636 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-02-18T00:22:55.919 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-18T00:22:55.919 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-18T00:22:56.933 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-02-18T00:22:56.947 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-02-18T00:22:57.163 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-18T00:22:57.186 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-18T00:22:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-02-18T00:22:59.985 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-18T00:23:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-18T00:23:00.520 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-18T00:23:04.070 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-18T00:23:04.091 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-18T00:23:04.469 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-18T00:23:04.511 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-18T00:23:05.039 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-18T00:23:05.040 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-18T00:23:05.184 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-02-18T00:23:05.185 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-02-18T00:23:05.185 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-02-18T00:23:05.185 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-02-18T00:23:05.185 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-02-18T00:23:05.185 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-18T00:23:05.185 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-02-18T00:23:05.186 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-18T00:23:05.186 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-18T00:23:05.186 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-02-18T00:23:05.186 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-02-18T00:23:05.186 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-02-18T00:23:05.186 INFO:teuthology.orchestra.run.smithi151.stdout:Install 3 Packages 2024-02-18T00:23:05.186 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-02-18T00:23:05.187 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 90 M 2024-02-18T00:23:05.187 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 97 M 2024-02-18T00:23:05.187 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-02-18T00:23:06.517 INFO:teuthology.orchestra.run.smithi151.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 8.0 MB/s | 10 MB 00:01 2024-02-18T00:23:07.914 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-18T00:23:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-18T00:23:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:08.459 INFO:teuthology.orchestra.run.smithi151.stdout:(2/3): kernel-core-4.18.0-539.el8.x86_64.rpm 13 MB/s | 43 MB 00:03 2024-02-18T00:23:08.496 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:09.425 INFO:teuthology.orchestra.run.smithi151.stdout:(3/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 8.6 MB/s | 36 MB 00:04 2024-02-18T00:23:09.426 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-02-18T00:23:09.426 INFO:teuthology.orchestra.run.smithi151.stdout:Total 21 MB/s | 90 MB 00:04 2024-02-18T00:23:10.109 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-02-18T00:23:10.571 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-02-18T00:23:10.571 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-02-18T00:23:11.841 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-02-18T00:23:11.888 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-02-18T00:23:14.774 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-02-18T00:23:15.295 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-18T00:23:18.865 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-18T00:23:19.278 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-18T00:23:22.723 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-18T00:23:23.051 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:27.477 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-18T00:23:27.477 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-18T00:23:28.041 INFO:teuthology.orchestra.run.smithi193.stderr:FATAL ERROR: python callback ??? failed, aborting! 2024-02-18T00:23:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-18T00:23:28.051 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/kernel.py", line 1238, in task p.spawn(process_role, ctx, config, timeout, role, role_config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/kernel.py", line 1270, in process_role version = need_to_install_distro(role_remote, role_config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/kernel.py", line 761, in need_to_install_distro install_stdout = remote.sh( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 1: 'sudo yum install -y kernel' 2024-02-18T00:23:28.282 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4936c643ff2d4fd4a28906ad28e57dbc Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/kernel.py", line 1238, in task p.spawn(process_role, ctx, config, timeout, role, role_config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/kernel.py", line 1270, in process_role version = need_to_install_distro(role_remote, role_config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/kernel.py", line 761, in need_to_install_distro install_stdout = remote.sh( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 1: 'sudo yum install -y kernel' 2024-02-18T00:23:28.285 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-18T00:23:28.296 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-18T00:23:28.379 INFO:teuthology.nuke:Checking targets against current locks 2024-02-18T00:23:28.428 DEBUG:teuthology.nuke:shortname: smithi038 2024-02-18T00:23:28.429 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-18T00:23:28.454 DEBUG:teuthology.nuke:shortname: smithi151 2024-02-18T00:23:28.454 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-18T00:23:28.460 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-18 00:17:12.931580', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLU4NWa2JJ9RrHp2d5NlizlOoZ3xs1JfjW88RxRBqdYPCMd4mZA/5Hlf0vBcqJilulqmRvkVBaSnFglKh0RawBI='} 2024-02-18T00:23:28.480 DEBUG:teuthology.nuke:shortname: smithi184 2024-02-18T00:23:28.480 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-18T00:23:28.488 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564973', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-18 00:17:13.390151', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEn6z+awfxqplhzV//lO7go10/J54r1co/CrvVwHkXYmxT4wqfDyrskZBIs8rc2FamGpS7erTo/qGB+3umrhI1U='} 2024-02-18T00:23:28.504 DEBUG:teuthology.nuke:shortname: smithi193 2024-02-18T00:23:28.507 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-18T00:23:28.522 INFO:teuthology.orchestra.console.smithi038:Power off 2024-02-18T00:23:28.522 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-18T00:23:28.530 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-18 00:17:12.934478', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf/R4z+6X5OwmVZAQTBGfm+VQadRCLX1Pq2P+G7sZDjfqZka+6QQ0tsPQ74jW7eAMAGz2A1U1X+bLfYUWobOx0='} 2024-02-18T00:23:28.536 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-17_15:55:51-rados-pacific-release-distro-default-smithi/7564973', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-18 00:17:13.388302', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBON54F23dLfDwlQHxNl9IVQ5sgUEuhnSFmGcT57OzIElOaVQjyxJ4dwcqvwqxZbS7KuMyG8wz6GRcDOT8DTbLdU='} 2024-02-18T00:23:28.546 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-02-18T00:23:28.546 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:28.562 INFO:teuthology.orchestra.console.smithi151:Power off 2024-02-18T00:23:28.563 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-18T00:23:28.581 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-02-18T00:23:28.587 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-02-18T00:23:28.587 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:28.611 INFO:teuthology.orchestra.console.smithi184:Power off 2024-02-18T00:23:28.612 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-18T00:23:28.625 INFO:teuthology.orchestra.console.smithi193:Power off 2024-02-18T00:23:28.625 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-18T00:23:28.634 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-02-18T00:23:28.634 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-02-18T00:23:28.634 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:28.648 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-02-18T00:23:28.649 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:28.661 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-02-18T00:23:28.671 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-02-18T00:23:30.183 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:30.205 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:30.534 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:30.534 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-18T00:23:30.535 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-18T00:23:31.065 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:31.065 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-18T00:23:31.065 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-18T00:23:31.066 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-18T00:23:31.066 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-18T00:23:31.066 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-18T00:23:31.066 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-18T00:23:31.204 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-02-18T00:23:31.464 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-18T00:23:31.464 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-18T00:23:31.464 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-18T00:23:31.464 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-18T00:23:31.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-18T00:23:31.465 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-02-18T00:23:31.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-02-18T00:23:31.465 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-02-18T00:23:31.674 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-18T00:23:31.675 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-18T00:23:31.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2024-02-18T00:23:31.709 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2024-02-18T00:23:31.709 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2024-02-18T00:23:31.709 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.02 2024-02-18T00:23:31.709 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 129.el8 2024-02-18T00:23:31.709 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2024-02-18T00:23:31.709 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-18T00:23:31.710 INFO:teuthology.orchestra.run.smithi184.stdout:Group : System Environment/Base 2024-02-18T00:23:31.710 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 9265586 2024-02-18T00:23:31.710 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2024-02-18T00:23:31.710 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-18T00:23:31.710 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-18T00:23:31.710 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-18T00:23:31.710 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-18T00:23:31.710 INFO:teuthology.orchestra.run.smithi184.stdout:Relocations : (not relocatable) 2024-02-18T00:23:31.711 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : CentOS BuildSystem 2024-02-18T00:23:31.711 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2024-02-18T00:23:31.711 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-18T00:23:31.711 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2024-02-18T00:23:31.711 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2024-02-18T00:23:31.711 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-18T00:23:31.711 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-18T00:23:31.711 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-18T00:23:31.712 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-18T00:23:31.712 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2024-02-18T00:23:31.712 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-18T00:23:31.712 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2024-02-18T00:23:31.713 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-18T00:23:31.713 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 4.18.0-539.el8.x86_64 2024-02-18T00:23:31.713 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-18T00:23:31.973 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:31.996 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:31.996 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-18T00:23:32.485 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:32.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-18T00:23:32.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-18T00:23:32.583 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:32.635 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:32.661 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:32.673 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:32.692 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-02-18T00:23:32.746 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-02-18T00:23:32.772 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-02-18T00:23:32.784 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-02-18T00:23:32.996 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-18T00:23:32.996 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-18T00:23:32.996 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-02-18T00:23:32.996 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-18T00:23:32.999 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-18T00:23:32.999 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-18T00:23:32.999 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-02-18T00:23:33.150 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-02-18T00:23:33.380 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-18T00:23:33.381 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-18T00:23:33.381 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-18T00:23:33.381 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-18T00:23:33.381 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-18T00:23:33.381 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-18T00:23:33.381 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-02-18T00:23:33.382 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-02-18T00:23:33.602 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-18T00:23:33.603 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-18T00:23:33.603 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-02-18T00:23:33.636 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-02-18T00:23:33.637 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-02-18T00:23:33.637 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.02 2024-02-18T00:23:33.637 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 129.el8 2024-02-18T00:23:33.637 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-02-18T00:23:33.637 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-18T00:23:33.637 INFO:teuthology.orchestra.run.smithi038.stdout:Group : System Environment/Base 2024-02-18T00:23:33.638 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 9265586 2024-02-18T00:23:33.638 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-02-18T00:23:33.638 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-18T00:23:33.638 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-18T00:23:33.638 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-18T00:23:33.638 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-18T00:23:33.638 INFO:teuthology.orchestra.run.smithi038.stdout:Relocations : (not relocatable) 2024-02-18T00:23:33.638 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : CentOS BuildSystem 2024-02-18T00:23:33.638 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-02-18T00:23:33.639 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-18T00:23:33.639 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-02-18T00:23:33.639 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-02-18T00:23:33.639 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-18T00:23:33.639 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-18T00:23:33.639 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-18T00:23:33.639 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-18T00:23:33.639 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-02-18T00:23:33.640 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-02-18T00:23:33.640 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-02-18T00:23:33.641 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-18T00:23:33.641 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 4.18.0-539.el8.x86_64 2024-02-18T00:23:33.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-18T00:23:35.170 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-02-18T00:23:36.693 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:36.747 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:36.773 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:36.784 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-18T00:23:36.804 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-02-18T00:23:36.858 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-02-18T00:23:36.885 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-02-18T00:23:36.894 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-02-18T00:23:36.905 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-02-18T00:23:36.959 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-02-18T00:23:36.986 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-02-18T00:23:36.995 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-02-18T00:23:37.100 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} failure_reason: 'Command failed on smithi193 with status 1: ''sudo yum install -y kernel''' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4936c643ff2d4fd4a28906ad28e57dbc status: fail success: false 2024-02-18T00:23:37.100 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-18T00:23:37.200 INFO:teuthology.run:FAIL