2024-04-10T20:24:41.071 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T20:24:41.071 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} --owner scheduled_yuriw@teuthology --verbose --name yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166 -- /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166/orig.config.yaml 2024-04-10T20:24:41.108 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T20:24:41.198 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166 branch: wip-yuri10-testing-2024-03-24-1159 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} email: yweinste@redhat.com first_in_suite: false job_id: '7651166' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'librbd C and C++ api tests ' name: yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-03-24-1159 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 filestore odsync write: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd max backfills: 2 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 0.5 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-yuri10-testing-2024-03-24-1159 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 8148 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-03-24-1159 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhR+bUlWWEnKxhf6Mefuwf7xwhu5fNDiqRg1wVrTrikpN97EfN9Gj6eSFr4SK010VU4qslu0ULvXovMrmpKfy0= smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBWn4QEhzGse21XN5oNWdB4BOQddXq6yzm6iumA018IahLZ2vciik6M64UBbDBOLymQHllnvk0kVMux83KDAgqQ= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFRk5ddEe2DM3s3lBYr4ArLhDLettH3tBqT1KUzBgl+wNh0Lse35brO2VV9bwFxacVMDVz4d8t8h6MVGefi3ssk= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: octopus exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 2 chance_pgpnum_fix: 1 timeout: 1200 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:20:47 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload: - workunit: clients: client.2: - rbd/test_librbd.sh 2024-04-10T20:24:41.198 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa; will attempt to use it 2024-04-10T20:24:41.199 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks 2024-04-10T20:24:41.199 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T20:24:41.201 INFO:teuthology.task.internal:Saving configuration 2024-04-10T20:24:41.215 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T20:24:41.221 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:24:41.256 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:19:37.912241', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhR+bUlWWEnKxhf6Mefuwf7xwhu5fNDiqRg1wVrTrikpN97EfN9Gj6eSFr4SK010VU4qslu0ULvXovMrmpKfy0='} 2024-04-10T20:24:41.291 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:19:37.910111', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBWn4QEhzGse21XN5oNWdB4BOQddXq6yzm6iumA018IahLZ2vciik6M64UBbDBOLymQHllnvk0kVMux83KDAgqQ='} 2024-04-10T20:24:41.323 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:19:37.911469', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFRk5ddEe2DM3s3lBYr4ArLhDLettH3tBqT1KUzBgl+wNh0Lse35brO2VV9bwFxacVMDVz4d8t8h6MVGefi3ssk='} 2024-04-10T20:24:41.323 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T20:24:41.329 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-10T20:24:41.329 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-10T20:24:41.330 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-10T20:24:41.330 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T20:24:41.469 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f76212c5f70>, signals=[15]) 2024-04-10T20:24:41.470 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T20:24:41.474 INFO:teuthology.task.internal:Opening connections... 2024-04-10T20:24:41.474 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-04-10T20:24:41.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:24:41.566 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2024-04-10T20:24:41.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:24:41.646 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2024-04-10T20:24:41.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:24:41.735 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T20:24:41.742 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-04-10T20:24:41.765 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-04-10T20:24:41.766 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-04-10T20:24:41.827 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-04-10T20:24:41.827 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="8" 2024-04-10T20:24:41.827 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-04-10T20:24:41.827 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-04-10T20:24:41.828 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="8" 2024-04-10T20:24:41.828 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el8" 2024-04-10T20:24:41.828 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T20:24:41.828 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-04-10T20:24:41.828 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T20:24:41.828 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-04-10T20:24:41.828 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T20:24:41.828 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T20:24:41.828 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T20:24:41.829 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-04-10T20:24:41.863 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2024-04-10T20:24:41.885 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2024-04-10T20:24:41.885 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T20:24:41.946 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T20:24:41.947 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T20:24:41.947 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2024-04-10T20:24:41.980 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2024-04-10T20:24:42.002 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2024-04-10T20:24:42.003 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2024-04-10T20:24:42.063 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2024-04-10T20:24:42.063 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="8" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="8" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el8" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T20:24:42.064 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T20:24:42.064 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-04-10T20:24:42.098 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T20:24:42.108 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T20:24:42.114 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T20:24:42.114 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-04-10T20:24:42.116 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2024-04-10T20:24:42.119 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2024-04-10T20:24:42.138 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T20:24:42.144 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T20:24:42.145 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-04-10T20:24:42.178 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2024-04-10T20:24:42.180 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2024-04-10T20:24:42.217 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T20:24:42.325 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T20:24:42.339 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T20:24:42.339 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T20:24:42.339 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T20:24:42.340 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T20:24:42.340 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T20:24:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:24:42.360 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-04-10T20:24:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:24:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:24:42.362 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-04-10T20:24:42.363 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-04-10T20:24:42.420 INFO:teuthology.orchestra.run.smithi167.stdout:4.18.0-489.el8.x86_64 2024-04-10T20:24:42.420 INFO:teuthology.task.kernel:Running kernel on smithi167: 4.18.0-489.el8.x86_64 2024-04-10T20:24:42.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2024-04-10T20:24:42.424 INFO:teuthology.orchestra.run.smithi149.stdout:4.18.0-489.el8.x86_64 2024-04-10T20:24:42.424 INFO:teuthology.task.kernel:Running kernel on smithi149: 4.18.0-489.el8.x86_64 2024-04-10T20:24:42.424 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum install -y kernel 2024-04-10T20:24:42.425 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-489.el8.x86_64 2024-04-10T20:24:42.425 INFO:teuthology.task.kernel:Running kernel on smithi050: 4.18.0-489.el8.x86_64 2024-04-10T20:24:42.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2024-04-10T20:24:46.165 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 8 - AppStream 22 MB/s | 28 MB 00:01 2024-04-10T20:26:56.446 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - AppStream 217 kB/s | 28 MB 02:11 2024-04-10T20:26:58.339 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - AppStream 214 kB/s | 28 MB 02:13 2024-04-10T20:27:05.387 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - BaseOS 2.9 MB/s | 10 MB 00:03 2024-04-10T20:27:05.388 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 8 - BaseOS 79 kB/s | 10 MB 02:14 2024-04-10T20:27:51.335 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 8 - Extras 411 B/s | 18 kB 00:44 2024-04-10T20:27:51.769 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 7.7 kB 00:00 2024-04-10T20:29:13.984 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - BaseOS 81 kB/s | 10 MB 02:10 2024-04-10T20:29:18.412 INFO:teuthology.orchestra.run.smithi167.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:26 2024-04-10T20:29:18.413 INFO:teuthology.orchestra.run.smithi167.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T20:29:18.413 INFO:teuthology.orchestra.run.smithi167.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T20:29:18.413 INFO:teuthology.orchestra.run.smithi167.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T20:29:19.649 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-04-10T20:29:23.447 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 410 kB/s | 24 kB 00:00 2024-04-10T20:29:23.593 INFO:teuthology.orchestra.run.smithi167.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T20:29:26.421 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T20:29:26.421 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T20:29:26.530 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-10T20:29:26.531 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-10T20:29:26.532 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-10T20:29:26.532 INFO:teuthology.orchestra.run.smithi167.stdout:Install 3 Packages 2024-04-10T20:29:26.532 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-10T20:29:26.532 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 90 M 2024-04-10T20:29:26.532 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 96 M 2024-04-10T20:29:26.532 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-10T20:29:29.625 INFO:teuthology.orchestra.run.smithi167.stdout:(1/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2024-04-10T20:29:30.734 INFO:teuthology.orchestra.run.smithi167.stdout:(2/3): kernel-4.18.0-552.el8.x86_64.rpm 2.6 MB/s | 10 MB 00:04 2024-04-10T20:29:54.741 INFO:teuthology.orchestra.run.smithi167.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 1.6 MB/s | 44 MB 00:28 2024-04-10T20:29:54.741 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-10T20:29:54.741 INFO:teuthology.orchestra.run.smithi167.stdout:Total 3.2 MB/s | 90 MB 00:28 2024-04-10T20:29:55.421 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-10T20:29:55.863 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-10T20:29:55.863 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-10T20:29:57.119 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-10T20:29:57.145 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-10T20:29:57.522 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - Extras 434 B/s | 18 kB 00:41 2024-04-10T20:29:57.981 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 7.7 kB 00:00 2024-04-10T20:29:59.952 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-10T20:30:00.551 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:30:03.724 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - Extras 103 B/s | 18 kB 02:56 2024-04-10T20:30:04.118 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:30:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:30:07.960 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:30:08.243 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:30:30.455 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:30:30.479 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:30:30.838 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:30:30.838 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:30:30.839 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:30:31.289 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:30:31.289 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-10T20:30:31.289 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-10T20:30:31.289 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:30:31.290 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:30:31.290 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-10T20:30:31.290 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-10T20:30:31.440 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:30:31.657 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:30:31.658 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:30:31.658 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:30:31.658 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:30:31.658 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:30:31.658 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-10T20:30:31.659 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:30:31.659 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:30:31.869 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:30:31.869 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:30:31.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo rpm -qi grub2-tools 2024-04-10T20:30:31.903 INFO:teuthology.orchestra.run.smithi167.stdout:Name : grub2-tools 2024-04-10T20:30:31.903 INFO:teuthology.orchestra.run.smithi167.stdout:Epoch : 1 2024-04-10T20:30:31.903 INFO:teuthology.orchestra.run.smithi167.stdout:Version : 2.02 2024-04-10T20:30:31.903 INFO:teuthology.orchestra.run.smithi167.stdout:Release : 129.el8 2024-04-10T20:30:31.903 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: x86_64 2024-04-10T20:30:31.903 INFO:teuthology.orchestra.run.smithi167.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:30:31.903 INFO:teuthology.orchestra.run.smithi167.stdout:Group : System Environment/Base 2024-04-10T20:30:31.903 INFO:teuthology.orchestra.run.smithi167.stdout:Size : 9265586 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:License : GPLv3+ 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:Relocations : (not relocatable) 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:Packager : CentOS BuildSystem 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:Vendor : CentOS 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:Summary : Support tools for GRUB. 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:Description : 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:30:31.904 INFO:teuthology.orchestra.run.smithi167.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:30:31.905 INFO:teuthology.orchestra.run.smithi167.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:30:31.905 INFO:teuthology.orchestra.run.smithi167.stdout:hardware devices. 2024-04-10T20:30:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-10T20:30:31.905 INFO:teuthology.orchestra.run.smithi167.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:30:31.906 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:30:31.906 INFO:teuthology.task.kernel:Updating grub on smithi167 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:30:31.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:30:33.372 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2024-04-10T20:30:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:done 2024-04-10T20:30:35.646 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-04-10T20:30:35.678 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.YIjK1cd6Bb 2024-04-10T20:30:35.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YIjK1cd6Bb 2024-04-10T20:30:35.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.YIjK1cd6Bb 2024-04-10T20:30:35.897 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.YIjK1cd6Bb is 6KB 2024-04-10T20:30:35.945 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.YIjK1cd6Bb 2024-04-10T20:30:35.962 DEBUG:teuthology.orchestra.run.smithi167:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:30:36.053 INFO:teuthology.orchestra.run.smithi167.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:30:36.053 INFO:teuthology.orchestra.run.smithi167.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:30:36.053 INFO:teuthology.orchestra.run.smithi167.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:30:36.053 INFO:teuthology.orchestra.run.smithi167.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:30:36.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:30:36.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2024-04-10T20:31:06.180 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-10T20:31:06.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-10T20:31:06.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:31:25.831 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:27 2024-04-10T20:31:25.832 INFO:teuthology.orchestra.run.smithi149.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T20:31:25.832 INFO:teuthology.orchestra.run.smithi149.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T20:31:25.832 INFO:teuthology.orchestra.run.smithi149.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T20:31:27.556 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 16 MB 00:01 2024-04-10T20:31:31.290 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 578 kB/s | 24 kB 00:00 2024-04-10T20:31:31.433 INFO:teuthology.orchestra.run.smithi149.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T20:31:34.229 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T20:31:34.229 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T20:31:34.338 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-10T20:31:34.339 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-10T20:31:34.340 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-10T20:31:34.340 INFO:teuthology.orchestra.run.smithi149.stdout:Install 3 Packages 2024-04-10T20:31:34.340 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-10T20:31:34.340 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 90 M 2024-04-10T20:31:34.340 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 96 M 2024-04-10T20:31:34.340 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-10T20:31:40.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-10T20:31:49.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-10T20:31:49.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:32:08.223 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-10T20:32:15.079 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - Extras common packages 60 B/s | 7.7 kB 02:11 2024-04-10T20:32:20.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-10T20:32:20.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:32:20.632 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-04-10T20:32:21.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-10T20:32:21.069 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:32:21.069 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-04-10T20:32:21.086 INFO:teuthology.orchestra.run.smithi167.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:32:21.087 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:32:21.087 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:32:21.087 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:32:22.087 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-10T20:32:22.088 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:32:22.174 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2024-04-10T20:32:22.189 DEBUG:teuthology.parallel:result is None 2024-04-10T20:33:43.540 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 01:28 2024-04-10T20:33:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2024-04-10T20:33:43.541 INFO:teuthology.orchestra.run.smithi050.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2024-04-10T20:33:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2024-04-10T20:33:44.767 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-04-10T20:33:48.781 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 607 kB/s | 24 kB 00:00 2024-04-10T20:33:48.932 INFO:teuthology.orchestra.run.smithi050.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2024-04-10T20:33:50.600 INFO:teuthology.orchestra.run.smithi149.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 3.2 MB/s | 10 MB 00:03 2024-04-10T20:33:52.045 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T20:33:52.045 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T20:33:52.167 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-04-10T20:33:52.168 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-04-10T20:33:52.168 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-04-10T20:33:52.168 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout:Install 3 Packages 2024-04-10T20:33:52.169 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-10T20:33:52.170 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 90 M 2024-04-10T20:33:52.170 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 96 M 2024-04-10T20:33:52.170 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-04-10T20:34:01.978 INFO:teuthology.orchestra.run.smithi149.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 3.0 MB/s | 44 MB 00:14 2024-04-10T20:34:04.640 INFO:teuthology.orchestra.run.smithi149.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 2.1 MB/s | 36 MB 00:17 2024-04-10T20:34:04.640 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-10T20:34:04.640 INFO:teuthology.orchestra.run.smithi149.stdout:Total 616 kB/s | 90 MB 02:30 2024-04-10T20:34:05.324 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-10T20:34:05.759 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-10T20:34:05.759 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-10T20:34:07.025 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-10T20:34:07.054 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-10T20:34:09.884 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-10T20:34:10.417 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:34:14.015 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:34:14.383 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:34:17.837 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:34:18.124 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:34:40.016 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:34:40.043 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:34:40.407 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:34:40.407 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:34:40.407 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:34:40.921 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:34:40.921 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-10T20:34:40.921 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-10T20:34:40.921 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:34:40.921 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:34:40.921 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-10T20:34:40.921 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-10T20:34:41.083 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:34:41.309 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:34:41.309 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:34:41.310 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:34:41.310 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:34:41.310 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:34:41.310 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-10T20:34:41.310 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi149.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:34:41.310 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:34:41.527 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:34:41.527 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:34:41.527 DEBUG:teuthology.orchestra.run.smithi149:> sudo rpm -qi grub2-tools 2024-04-10T20:34:41.561 INFO:teuthology.orchestra.run.smithi149.stdout:Name : grub2-tools 2024-04-10T20:34:41.561 INFO:teuthology.orchestra.run.smithi149.stdout:Epoch : 1 2024-04-10T20:34:41.561 INFO:teuthology.orchestra.run.smithi149.stdout:Version : 2.02 2024-04-10T20:34:41.561 INFO:teuthology.orchestra.run.smithi149.stdout:Release : 129.el8 2024-04-10T20:34:41.561 INFO:teuthology.orchestra.run.smithi149.stdout:Architecture: x86_64 2024-04-10T20:34:41.561 INFO:teuthology.orchestra.run.smithi149.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Group : System Environment/Base 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Size : 9265586 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:License : GPLv3+ 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Relocations : (not relocatable) 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Packager : CentOS BuildSystem 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Vendor : CentOS 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Summary : Support tools for GRUB. 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout:Description : 2024-04-10T20:34:41.562 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-10T20:34:41.563 INFO:teuthology.orchestra.run.smithi149.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:34:41.563 INFO:teuthology.orchestra.run.smithi149.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:34:41.563 INFO:teuthology.orchestra.run.smithi149.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:34:41.563 INFO:teuthology.orchestra.run.smithi149.stdout:hardware devices. 2024-04-10T20:34:41.563 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-10T20:34:41.563 INFO:teuthology.orchestra.run.smithi149.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:34:41.564 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:34:41.564 INFO:teuthology.task.kernel:Updating grub on smithi149 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:34:41.564 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:34:43.170 INFO:teuthology.orchestra.run.smithi149.stderr:Generating grub configuration file ... 2024-04-10T20:34:45.274 INFO:teuthology.orchestra.run.smithi149.stderr:done 2024-04-10T20:34:45.275 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2024-04-10T20:34:45.298 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.vKCfmeIdOD 2024-04-10T20:34:45.298 DEBUG:teuthology.orchestra.run.smithi149:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vKCfmeIdOD 2024-04-10T20:34:45.372 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.vKCfmeIdOD 2024-04-10T20:34:45.514 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.vKCfmeIdOD is 6KB 2024-04-10T20:34:45.561 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.vKCfmeIdOD 2024-04-10T20:34:45.579 DEBUG:teuthology.orchestra.run.smithi149:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:34:45.651 INFO:teuthology.orchestra.run.smithi149.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:34:45.651 INFO:teuthology.orchestra.run.smithi149.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:34:45.651 INFO:teuthology.orchestra.run.smithi149.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:34:45.651 INFO:teuthology.orchestra.run.smithi149.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:34:45.652 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:34:45.807 DEBUG:teuthology.orchestra.run.smithi149:> sudo shutdown -r now 2024-04-10T20:35:15.811 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-10T20:35:15.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-04-10T20:35:15.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:35:50.431 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-04-10T20:35:59.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-04-10T20:35:59.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:36:12.211 INFO:teuthology.orchestra.run.smithi050.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 1.3 MB/s | 10 MB 00:07 2024-04-10T20:36:17.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-04-10T20:36:18.826 INFO:teuthology.orchestra.run.smithi050.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 3.0 MB/s | 44 MB 00:14 2024-04-10T20:36:29.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-04-10T20:36:29.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:36:30.345 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-04-10T20:36:30.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-04-10T20:36:30.629 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:36:30.629 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-04-10T20:36:30.684 INFO:teuthology.orchestra.run.smithi149.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:36:30.684 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:36:30.684 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:36:30.685 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:36:31.686 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-10T20:36:31.686 DEBUG:teuthology.orchestra.run.smithi149:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:36:31.805 INFO:teuthology.orchestra.run.smithi149.stdout:ttyS1 2024-04-10T20:36:31.817 DEBUG:teuthology.parallel:result is None 2024-04-10T20:36:32.351 INFO:teuthology.orchestra.run.smithi050.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 1.3 MB/s | 36 MB 00:28 2024-04-10T20:36:32.351 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-04-10T20:36:32.351 INFO:teuthology.orchestra.run.smithi050.stdout:Total 578 kB/s | 90 MB 02:40 2024-04-10T20:36:33.064 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-04-10T20:36:33.528 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-04-10T20:36:33.528 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-04-10T20:36:34.848 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-04-10T20:36:34.880 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-04-10T20:36:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-04-10T20:36:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:36:42.206 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:36:42.611 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:36:46.296 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:36:46.614 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:37:10.108 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:37:10.133 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:37:10.497 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:37:10.497 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T20:37:10.497 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T20:37:11.051 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T20:37:11.051 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-10T20:37:11.051 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-04-10T20:37:11.051 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T20:37:11.051 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T20:37:11.051 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-10T20:37:11.052 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-04-10T20:37:11.203 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:37:11.446 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:37:11.447 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:37:11.447 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T20:37:11.447 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T20:37:11.447 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T20:37:11.447 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T20:37:11.447 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T20:37:11.447 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T20:37:11.668 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T20:37:11.668 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T20:37:11.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2024-04-10T20:37:11.702 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2024-04-10T20:37:11.702 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2024-04-10T20:37:11.702 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.02 2024-04-10T20:37:11.702 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 129.el8 2024-04-10T20:37:11.702 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Group : System Environment/Base 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 9265586 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Relocations : (not relocatable) 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : CentOS BuildSystem 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2024-04-10T20:37:11.703 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2024-04-10T20:37:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-10T20:37:11.704 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T20:37:11.704 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T20:37:11.704 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T20:37:11.704 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2024-04-10T20:37:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-10T20:37:11.704 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T20:37:11.705 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T20:37:11.705 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 4.18.0-552.el8.x86_64 2024-04-10T20:37:11.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T20:37:13.019 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2024-04-10T20:37:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:done 2024-04-10T20:37:15.103 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-04-10T20:37:15.128 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.BiDpWTdp50 2024-04-10T20:37:15.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BiDpWTdp50 2024-04-10T20:37:15.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.BiDpWTdp50 2024-04-10T20:37:15.363 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.BiDpWTdp50 is 6KB 2024-04-10T20:37:15.412 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.BiDpWTdp50 2024-04-10T20:37:15.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T20:37:15.509 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T20:37:15.509 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T20:37:15.509 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T20:37:15.509 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T20:37:15.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T20:37:15.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2024-04-10T20:37:45.674 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T20:37:45.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-10T20:37:45.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:38:45.706 DEBUG:teuthology.orchestra.remote:timed out 2024-04-10T20:38:54.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-10T20:38:54.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T20:38:55.011 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-04-10T20:38:55.311 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-10T20:38:55.311 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T20:38:55.312 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-04-10T20:38:55.329 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-552.el8.x86_64 2024-04-10T20:38:55.329 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T20:38:55.329 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T20:38:55.329 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T20:38:56.330 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T20:38:56.330 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T20:38:56.396 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-04-10T20:38:56.408 DEBUG:teuthology.parallel:result is None 2024-04-10T20:38:56.408 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T20:38:56.419 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T20:38:56.419 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:38:56.422 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:38:56.426 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T20:38:56.451 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T20:38:56.460 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T20:38:56.473 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T20:38:56.474 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:38:56.484 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:38:56.493 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T20:38:56.538 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T20:38:56.548 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T20:38:56.549 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:38:56.557 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:38:56.576 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T20:38:56.591 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:38:56.602 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:38:56.618 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:38:56.626 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:38:56.629 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:38:56.638 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T20:38:56.639 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T20:38:56.647 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T20:38:56.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:38:56.651 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:38:56.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T20:38:56.703 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T20:38:56.714 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T20:38:56.714 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:38:56.721 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:38:56.743 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T20:38:56.763 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:38:56.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:38:56.879 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:38:56.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:38:57.008 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-10T20:38:57.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:38:57.077 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:38:57.135 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:38:57.184 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:38:57.240 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:38:57.274 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-10T20:38:57.275 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:38:57.343 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:38:57.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T20:38:57.455 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:38:57.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T20:38:57.546 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-10T20:38:57.546 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T20:38:57.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-04-10T20:38:57.617 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2024-04-10T20:38:57.620 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2024-04-10T20:38:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:38:57.690 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:38:57.724 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:38:58.156 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T20:38:58.164 INFO:teuthology.task.internal:Starting timer... 2024-04-10T20:38:58.164 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T20:38:58.177 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T20:38:58.186 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T20:38:58.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-04-10T20:38:58.277 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-04-10T20:38:58.279 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2024-04-10T20:38:58.372 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2024-04-10T20:38:58.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo service auditd rotate 2024-04-10T20:38:58.471 INFO:teuthology.orchestra.run.smithi167.stdout:Rotating logs: 2024-04-10T20:38:58.473 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T20:38:58.473 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-04-10T20:38:58.501 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-04-10T20:38:58.501 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2024-04-10T20:38:58.575 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2024-04-10T20:38:58.575 DEBUG:teuthology.orchestra.run.smithi167:> /usr/sbin/getenforce 2024-04-10T20:38:58.597 INFO:teuthology.orchestra.run.smithi167.stdout:Permissive 2024-04-10T20:38:58.597 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi149.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi167.front.sepia.ceph.com': 'permissive'} 2024-04-10T20:38:58.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:38:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:38:58.626 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:38:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:38:58.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:38:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:38:58.682 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T20:38:58.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:38:58.712 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:38:58.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo /usr/sbin/setenforce permissive 2024-04-10T20:38:58.767 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T20:38:58.776 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T20:38:58.783 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T20:38:58.981 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T20:38:58.998 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T20:38:58.999 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi149.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T20:55:08.622 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166/ansible_failures.yaml 2024-04-10T20:55:08.625 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T20:55:08.839 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=495976b7496f449c923154d11edfc2c7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T20:55:08.841 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T20:55:08.849 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T20:55:08.850 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T20:55:08.899 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T20:55:08.981 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T20:55:09.044 DEBUG:teuthology.orchestra.run.smithi167:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T20:55:09.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:55:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:55:09.143 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:55:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:55:09.174 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T20:55:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:55:09.206 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T20:55:09.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T20:55:09.244 INFO:teuthology.task.internal:Duration was 971.079709 seconds 2024-04-10T20:55:09.244 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T20:55:09.269 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T20:55:09.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T20:55:09.272 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T20:55:09.274 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T20:55:09.321 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:55:09.323 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:55:09.324 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T20:55:09.783 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T20:55:09.783 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-04-10T20:55:09.783 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T20:55:09.812 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2024-04-10T20:55:09.812 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T20:55:09.839 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2024-04-10T20:55:09.839 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T20:55:09.864 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T20:55:09.864 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T20:55:09.867 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T20:55:09.882 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T20:55:09.946 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T20:55:09.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T20:55:09.949 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T20:55:09.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T20:55:10.050 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T20:55:10.059 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T20:55:10.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T20:55:10.090 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T20:55:10.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T20:55:10.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T20:55:10.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:55:10.156 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:55:10.160 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:55:10.182 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-04-10T20:55:10.184 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2024-04-10T20:55:10.211 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2024-04-10T20:55:10.232 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:55:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:55:10.260 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:55:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:55:10.277 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T20:55:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T20:55:10.294 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T20:55:10.302 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T20:55:10.303 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166/remote/smithi050 2024-04-10T20:55:10.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T20:55:10.365 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166/remote/smithi149 2024-04-10T20:55:10.366 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T20:55:10.448 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166/remote/smithi167 2024-04-10T20:55:10.449 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T20:55:10.507 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T20:55:10.507 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T20:55:10.509 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T20:55:10.512 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T20:55:10.529 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T20:55:10.539 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T20:55:10.539 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T20:55:10.565 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T20:55:10.565 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T20:55:10.569 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T20:55:10.571 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T20:55:10.585 INFO:teuthology.orchestra.run.smithi050.stdout: 525355 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 20:55 /home/ubuntu/cephtest 2024-04-10T20:55:10.587 INFO:teuthology.orchestra.run.smithi149.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 20:55 /home/ubuntu/cephtest 2024-04-10T20:55:10.589 INFO:teuthology.orchestra.run.smithi167.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 20:55 /home/ubuntu/cephtest 2024-04-10T20:55:10.590 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T20:55:10.598 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T20:55:10.708 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T20:55:10.746 DEBUG:teuthology.nuke:shortname: smithi050 2024-04-10T20:55:10.746 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:55:10.793 DEBUG:teuthology.nuke:shortname: smithi149 2024-04-10T20:55:10.793 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:55:10.821 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:19:37.912241', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhR+bUlWWEnKxhf6Mefuwf7xwhu5fNDiqRg1wVrTrikpN97EfN9Gj6eSFr4SK010VU4qslu0ULvXovMrmpKfy0='} 2024-04-10T20:55:10.840 DEBUG:teuthology.nuke:shortname: smithi167 2024-04-10T20:55:10.841 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T20:55:10.855 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:19:37.910111', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBWn4QEhzGse21XN5oNWdB4BOQddXq6yzm6iumA018IahLZ2vciik6M64UBbDBOLymQHllnvk0kVMux83KDAgqQ='} 2024-04-10T20:55:10.891 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:20:47-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7651166', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 20:19:37.911469', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFRk5ddEe2DM3s3lBYr4ArLhDLettH3tBqT1KUzBgl+wNh0Lse35brO2VV9bwFxacVMDVz4d8t8h6MVGefi3ssk='} 2024-04-10T20:55:10.911 INFO:teuthology.orchestra.console.smithi050:Power off 2024-04-10T20:55:10.912 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T20:55:10.960 INFO:teuthology.orchestra.console.smithi149:Power off 2024-04-10T20:55:10.961 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T20:55:10.983 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2024-04-10T20:55:10.984 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:11.010 INFO:teuthology.orchestra.console.smithi167:Power off 2024-04-10T20:55:11.011 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T20:55:11.019 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2024-04-10T20:55:11.019 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:11.034 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-04-10T20:55:11.060 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2024-04-10T20:55:11.060 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:11.069 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-04-10T20:55:11.110 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-04-10T20:55:15.035 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:15.069 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:15.111 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:15.145 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-04-10T20:55:15.179 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-04-10T20:55:15.221 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-04-10T20:55:19.150 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:19.180 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:19.222 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:19.261 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2024-04-10T20:55:19.288 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2024-04-10T20:55:19.330 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-04-10T20:55:19.362 INFO:teuthology.orchestra.console.smithi050:Power off completed 2024-04-10T20:55:19.388 INFO:teuthology.orchestra.console.smithi149:Power off completed 2024-04-10T20:55:23.336 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T20:55:23.445 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-04-10T20:55:23.546 INFO:teuthology.orchestra.console.smithi167:Power off completed 2024-04-10T20:55:23.652 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} duration: 971.0797090530396 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=495976b7496f449c923154d11edfc2c7 status: dead success: false 2024-04-10T20:55:23.652 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T20:55:23.751 INFO:teuthology.run:DEAD